Abstract
Multi-Agent Reinforcement Learning (MARL) has shown promise in solvingcomplex problems involving cooperation and competition among agents, such as anUnmanned Surface Vehicle (USV) swarm used in search and rescue, surveillance,and vessel protection. However, aligning system behavior with user preferencesis challenging due to the difficulty of encoding expert intuition into rewardfunctions. To address the issue, we propose a Reinforcement Learning with HumanFeedback (RLHF) approach for MARL that resolves credit-assignment challengesthrough an Agent-Level Feedback system categorizing feedback into intra-agent,inter-agent, and intra-team types. To overcome the challenges of direct humanfeedback, we employ a Large Language Model (LLM) evaluator to validate ourapproach using feedback scenarios such as region constraints, collisionavoidance, and task allocation. Our method effectively refines USV swarmpolicies, addressing key challenges in multi-agent systems while maintainingfairness and performance consistency.