Abstract
An increasing number of studies have focused on stochastic first-ordermethods (SFOMs) under heavy-tailed gradient noises, which have been observed inthe training of practical deep learning models. In this paper, we focus on twotypes of gradient noises: one is sub-Weibull noise, and the other is noiseunder the assumption that it has a bounded $p$-th central moment ($p$-BCM) with$p\in (1, 2]$. The latter is more challenging due to the occurrence of infinitevariance when $p\in (1, 2)$. Under these two gradient noise assumptions, thein-expectation and high-probability convergence of SFOMs have been extensivelystudied in the contexts of convex optimization and standard smoothoptimization. However, for weakly convex objectives-a class that includes allLipschitz-continuous convex objectives and smooth objectives-our understandingof the in-expectation and high-probability convergence of SFOMs under these twotypes of noises remains incomplete. We investigate the high-probabilityconvergence of the vanilla stochastic subgradient descent (SsGD) method undersub-Weibull noises, as well as the high-probability and in-expectationconvergence of clipped SsGD under the $p$-BCM noises. Both analyses areconducted in the context of weakly convex optimization. For weakly convexobjectives that may be non-convex and non-smooth, our results demonstrate thatthe theoretical dependence of vanilla SsGD on the failure probability andnumber of iterations under sub-Weibull noises does not degrade compared to thecase of smooth objectives. Under $p$-BCM noises, our findings indicate that thenon-smoothness and non-convexity of weakly convex objectives do not impact thetheoretical dependence of clipped SGD on the failure probability relative tothe smooth case; however, the sample complexity we derived is worse than awell-known lower bound for smooth optimization.