Abstract
The effect of relative entropy asymmetry is analyzed in the context ofempirical risk minimization (ERM) with relative entropy regularization(ERM-RER). Two regularizations are considered: $(a)$ the relative entropy ofthe measure to be optimized with respect to a reference measure (Type-IERM-RER); or $(b)$ the relative entropy of the reference measure with respectto the measure to be optimized (Type-II ERM-RER). The main result is thecharacterization of the solution to the Type-II ERM-RER problem and its keyproperties. By comparing the well-understood Type-I ERM-RER with Type-IIERM-RER, the effects of entropy asymmetry are highlighted. The analysis showsthat in both cases, regularization by relative entropy forces the solution'ssupport to collapse into the support of the reference measure, introducing astrong inductive bias that can overshadow the evidence provided by the trainingdata. Finally, it is shown that Type-II regularization is equivalent to Type-Iregularization with an appropriate transformation of the empirical riskfunction.