Abstract
Large language models (LLMs) are renowned for their extensive linguisticknowledge and strong generalization capabilities, but their high computationaldemands make them unsuitable for resource-constrained environments. Incontrast, small language models (SLMs) are computationally efficient but oftenlack the broad generalization capacity of LLMs. To bridge this gap, we proposePiFi, a novel framework that combines the strengths of both LLMs and SLMs toachieve high performance while maintaining efficiency. PiFi integrates a singlefrozen layer from an LLM into a SLM and fine-tunes the combined model forspecific tasks, boosting performance without a significant increase incomputational cost. We show that PiFi delivers consistent performanceimprovements across a range of natural language processing tasks, includingboth natural language understanding and generation. Moreover, our findingsdemonstrate PiFi's ability to effectively leverage LLM knowledge, enhancinggeneralization to unseen domains and facilitating the transfer of linguisticabilities.