Abstract
High-dose-rate (HDR) brachytherapy plays a critical role in the treatment oflocally advanced cervical cancer but remains highly dependent on manualtreatment planning expertise. The objective of this study is to develop a fullyautomated HDR brachytherapy planning framework that integrates reinforcementlearning (RL) and dose-based optimization to generate clinically acceptabletreatment plans with improved consistency and efficiency. We propose ahierarchical two-stage autoplanning framework. In the first stage, a deepQ-network (DQN)-based RL agent iteratively selects treatment planningparameters (TPPs), which control the trade-offs between target coverage andorgan-at-risk (OAR) sparing. The agent's state representation includes bothdose-volume histogram (DVH) metrics and current TPP values, while its rewardfunction incorporates clinical dose objectives and safety constraints,including D90, V150, V200 for targets, and D2cc for all relevant OARs (bladder,rectum, sigmoid, small bowel, and large bowel). In the second stage, acustomized Adam-based optimizer computes the corresponding dwell timedistribution for the selected TPPs using a clinically informed loss function.The framework was evaluated on a cohort of patients with complex applicatorgeometries. The proposed framework successfully learned clinically meaningfulTPP adjustments across diverse patient anatomies. For the unseen test patients,the RL-based automated planning method achieved an average score of 93.89%,outperforming the clinical plans which averaged 91.86%. These findings arenotable given that score improvements were achieved while maintaining fulltarget coverage and reducing CTV hot spots in most cases.