Abstract
Supervised fine-tuning (SFT) and alignment of large language models (LLMs)are key steps in providing a good user experience. However, the concept of anappropriate alignment is inherently application-dependent, and current methodsoften rely on heuristic choices to drive optimization. In this work, weformulate SFT and alignment as a constrained optimization problem: the LLM isfine-tuned on a task while being required to meet application-specificrequirements, without resorting to heuristics. To solve this, we proposeLagrange Large Language Models (L3Ms), which employ logarithmic barriers toenforce the constraints. This approach allows for the customization of L3Msacross diverse applications while avoiding heuristic-driven processes. Weexperimentally demonstrate the versatility and efficacy of L3Ms in achievingtailored alignments for various applications.