Abstract
Haptic captioning is the task of generating natural language descriptionsfrom haptic signals, such as vibrations, for use in virtual reality,accessibility, and rehabilitation applications. While previous multimodalresearch has focused primarily on vision and audio, haptic signals for thesense of touch remain underexplored. To address this gap, we formalize thehaptic captioning task and propose HapticLLaMA, a multimodal sensory languagemodel that interprets vibration signals into descriptions in a given sensory,emotional, or associative category. We investigate two types of haptictokenizers, a frequency-based tokenizer and an EnCodec-based tokenizer, thatconvert haptic signals into sequences of discrete units, enabling theirintegration with the LLaMA model. HapticLLaMA is trained in two stages: (1)supervised fine-tuning using the LLaMA architecture with LoRA-based adaptation,and (2) fine-tuning via reinforcement learning from human feedback (RLHF). Weassess HapticLLaMA's captioning performance using both automated n-gram metricsand human evaluation. HapticLLaMA demonstrates strong capability ininterpreting haptic vibration signals, achieving a METEOR score of 59.98 and aBLEU-4 score of 32.06 respectively. Additionally, over 61% of the generatedcaptions received human ratings above 3.5 on a 7-point scale, with RLHFyielding a 10% improvement in the overall rating distribution, indicatingstronger alignment with human haptic perception. These findings highlight thepotential of large language models to process and adapt to sensory data.