Abstract
Recent advances in the development of pre-trained Spanish language models hasled to significant progress in many Natural Language Processing (NLP) tasks,such as question answering. However, the lack of efficient models imposes abarrier for the adoption of such models in resource-constrained environments.Therefore, smaller distilled models for the Spanish language could be proven tobe highly scalable and facilitate their further adoption on a variety of tasksand scenarios. In this work, we take one step in this direction by developingSpanishTinyRoBERTa, a compressed language model based on RoBERTa for efficientquestion answering in Spanish. To achieve this, we employ knowledgedistillation from a large model onto a lighter model that allows for a widerimplementation, even in areas with limited computational resources, whilstattaining negligible performance sacrifice. Our experiments show that the densedistilled model can still preserve the performance of its larger counterpart,while significantly increasing inference speedup. This work serves as astarting point for further research and investigation of model compressionefforts for Spanish language models across various NLP tasks.