Abstract
We explore on-device self-supervised collaborative fine-tuning of largelanguage models with limited local data availability. Taking inspiration fromthe collaborative learning community, we introduce three distincttrust-weighted gradient aggregation schemes: weight similarity-based,prediction similarity-based and validation performance-based. To minimizecommunication overhead, we integrate Low-Rank Adaptation (LoRA) and onlyexchange LoRA weight updates. Our protocols, driven by prediction andperformance metrics, surpass both FedAvg and local fine-tuning methods, whichis particularly evident in realistic scenarios with more diverse local datadistributions. The results underscore the effectiveness of our approach inaddressing heterogeneity and scarcity within local datasets.