Abstract
Fine-tuning Large Language Models (LLMs) for clinical Natural LanguageProcessing (NLP) poses significant challenges due to domain gap, limited data,and stringent hardware constraints. In this study, we evaluate four adaptertechniques-Adapter, Lightweight, TinyAttention, and Gated Residual Network(GRN) - equivalent to Low-Rank Adaptation (LoRA), for clinical noteclassification under real-world, resource-constrained conditions. Allexperiments were conducted on a single NVIDIA Quadro P620 GPU (2 GB VRAM, 512CUDA cores, 1.386 TFLOPS FP32), limiting batch sizes to <8 sequences andmaximum sequence length to 256 tokens. Our clinical corpus comprises only 580000 tokens, several orders of magnitude smaller than standard LLM pre-trainingdatasets. We fine-tuned three biomedical pre-trained LLMs (CamemBERT-bio,AliBERT, DrBERT) and two lightweight Transformer models trained from scratch.Results show that 1) adapter structures provide no consistent gains whenfine-tuning biomedical LLMs under these constraints, and 2) simplerTransformers, with minimal parameter counts and training times under six hours,outperform adapter-augmented LLMs, which required over 1000 GPU-hours. Amongadapters, GRN achieved the best metrics (accuracy, precision, recall, F1 =0.88). These findings demonstrate that, in low-resource clinical settings withlimited data and compute, lightweight Transformers trained from scratch offer amore practical and efficient solution than large LLMs, while GRN remains aviable adapter choice when minimal adaptation is needed.