Abstract
Constrained optimization with multiple functional inequality constraints hassignificant applications in machine learning. This paper examines a crucialsubset of such problems where both the objective and constraint functions areweakly convex. Existing methods often face limitations, including slowconvergence rates or reliance on double-loop algorithmic designs. To overcomethese challenges, we introduce a novel single-loop penalty-based stochasticalgorithm. Following the classical exact penalty method, our approach employs a{\bf hinge-based penalty}, which permits the use of a constant penaltyparameter, enabling us to achieve a {\bf state-of-the-art complexity} forfinding an approximate Karush-Kuhn-Tucker (KKT) solution. We further extend ouralgorithm to address finite-sum coupled compositional objectives, which areprevalent in artificial intelligence applications, establishing improvedcomplexity over existing approaches. Finally, we validate our method throughexperiments on fair learning with receiver operating characteristic (ROC)fairness constraints and continual learning with non-forgetting constraints.