Abstract
The advancement of natural language processing (NLP) in biology hinges onmodels' ability to interpret intricate biomedical literature. Traditionalmodels often struggle with the complex and domain-specific language in thisfield. In this paper, we present BioMamba, a pre-trained model specificallydesigned for biomedical text mining. BioMamba builds upon the Mambaarchitecture and is pre-trained on an extensive corpus of biomedicalliterature. Our empirical studies demonstrate that BioMamba significantlyoutperforms models like BioBERT and general-domain Mamba across variousbiomedical tasks. For instance, BioMamba achieves a 100 times reduction inperplexity and a 4 times reduction in cross-entropy loss on the BioASQ testset. We provide an overview of the model architecture, pre-training process,and fine-tuning techniques. Additionally, we release the code and trained modelto facilitate further research.