Abstract
Surgical automation holds immense potential to improve the outcome andaccessibility of surgery. Recent studies use reinforcement learning to learnpolicies that automate different surgical tasks. However, these policies aredeveloped independently and are limited in their reusability when the taskchanges, making it more time-consuming when robots learn to solve multipletasks. Inspired by how human surgeons build their expertise, we train surgicalautomation policies through Surgical Incremental Reinforcement Learning(SurgIRL). SurgIRL aims to (1) acquire new skills by referring to externalpolicies (knowledge) and (2) accumulate and reuse these skills to solvemultiple unseen tasks incrementally (incremental learning). Our SurgIRLframework includes three major components. We first define an expandableknowledge set containing heterogeneous policies that can be helpful forsurgical tasks. Then, we propose Knowledge Inclusive Attention Network withmAximum Coverage Exploration (KIAN-ACE), which improves learning efficiency bymaximizing the coverage of the knowledge set during the exploration process.Finally, we develop incremental learning pipelines based on KIAN-ACE toaccumulate and reuse learned knowledge and solve multiple surgical taskssequentially. Our simulation experiments show that KIAN-ACE efficiently learnsto automate ten surgical tasks separately or incrementally. We also evaluateour learned policies on the da Vinci Research Kit (dVRK) and demonstratesuccessful sim-to-real transfers.