Abstract
Lipschitz continuity is a crucial functional property of any predictivemodel, that naturally governs its robustness, generalisation, as well asadversarial vulnerability. Contrary to other works that focus on obtainingtighter bounds and developing different practical strategies to enforce certainLipschitz properties, we aim to thoroughly examine and characterise theLipschitz behaviour of Neural Networks. Thus, we carry out an empiricalinvestigation in a range of different settings (namely, architectures,datasets, label noise, and more) by exhausting the limits of the simplest andthe most general lower and upper bounds. As a highlight of this investigation,we showcase a remarkable fidelity of the lower Lipschitz bound, identify astriking Double Descent trend in both upper and lower bounds to the Lipschitzand explain the intriguing effects of label noise on function smoothness andgeneralisation.