Abstract
Common Stochastic Gradient MCMC methods approximate gradients by stochasticones via uniformly subsampled data points. We propose that a non-uniformsubsampling can reduce the variance introduced by the stochastic approximation,hence making the sampling of a target distribution more accurate. Anexponentially weighted stochastic gradient approach (EWSG) is developed forthis objective by matching the transition kernels of SG-MCMC methodsrespectively based on stochastic and batch gradients. A demonstration of EWSGcombined with second-order Langevin equation for sampling purposes is provided.In our method, non-uniform subsampling is done efficiently via aMetropolis-Hasting chain on the data index, which is coupled to the samplingalgorithm. The fact that our method has reduced local variance with highprobability is theoretically analyzed. A non-asymptotic global error analysisis also presented. Numerical experiments based on both synthetic and real worlddata sets are also provided to demonstrate the efficacy of the proposedapproaches. While statistical accuracy has improved, the speed of convergencewas empirically observed to be at least comparable to the uniform version.