Abstract
Penalizing the nuclear norm of a function's Jacobian encourages it to locallybehave like a low-rank linear map. Such functions vary locally along only ahandful of directions, making the Jacobian nuclear norm a natural regularizerfor machine learning problems. However, this regularizer is intractable forhigh-dimensional problems, as it requires computing a large Jacobian matrix andtaking its singular value decomposition. We show how to efficiently penalizethe Jacobian nuclear norm using techniques tailor-made for deep learning. Weprove that for functions parametrized as compositions $f = g \circ h$, one mayequivalently penalize the average squared Frobenius norm of $Jg$ and $Jh$. Wethen propose a denoising-style approximation that avoids the Jacobiancomputations altogether. Our method is simple, efficient, and accurate,enabling Jacobian nuclear norm regularization to scale to high-dimensional deeplearning problems. We complement our theory with an empirical study of ourregularizer's performance and investigate applications to denoising andrepresentation learning.