Abstract
While scaling laws have been continuously validated in large language models(LLMs) with increasing model parameters, the inherent tension between theinference demands of LLMs and the limited resources of edge devices poses acritical challenge to the development of edge intelligence. Recently, numeroussmall language models have emerged, aiming to distill the capabilities of LLMsinto smaller footprints. However, these models often retain the fundamentalarchitectural principles of their larger counterparts, still imposingconsiderable strain on the storage and bandwidth capacities of edge devices. Inthis paper, we introduce the PLM, a Peripheral Language Model, developedthrough a co-design process that jointly optimizes model architecture and edgesystem constraints. The PLM utilizes a Multi-head Latent Attention mechanismand employs the squared ReLU activation function to encourage sparsity, therebyreducing peak memory footprint during inference. During training, we collectand reorganize open-source datasets, implement a multi-phase training strategy,and empirically investigate the Warmup-Stable-Decay-Constant (WSDC) learningrate scheduler. Additionally, we incorporate Reinforcement Learning from HumanFeedback (RLHF) by adopting the ARIES preference learning approach. Following atwo-phase SFT process, this method yields performance gains of 2% in generaltasks, 9% in the GSM8K task, and 11% in coding tasks. In addition to its novelarchitecture, evaluation results demonstrate that PLM outperforms existingsmall language models trained on publicly available data while maintaining thelowest number of activated parameters. Furthermore, deployment across variousedge devices, including consumer-grade GPUs, mobile phones, and Raspberry Pis,validates PLM's suitability for peripheral applications. The PLM series modelsare publicly available at https://github.com/plm-team/PLM.