Abstract
This paper derives a complete set of quadratic constraints (QCs) for therepeated ReLU. The complete set of QCs is described by a collection of matrixcopositivity conditions. We also show that only two functions satisfy all QCsin our complete set: the repeated ReLU and flipped ReLU. Thus our complete setof QCs bounds the repeated ReLU as tight as possible up to the sign invarianceinherent in quadratic forms. We derive a similar complete set of incrementalQCs for repeated ReLU, which can potentially lead to less conservativeLipschitz bounds for ReLU networks than the standard LipSDP approach. The basicconstructions are also used to derive the complete sets of QCs for otherpiecewise linear activation functions such as leaky ReLU, MaxMin, andHouseHolder. Finally, we illustrate the use of the complete set of QCs toassess stability and performance for recurrent neural networks with ReLUactivation functions. We rely on a standard copositivity relaxation toformulate the stability/performance condition as a semidefinite program. Simpleexamples are provided to illustrate that the complete sets of QCs andincremental QCs can yield less conservative bounds than existing sets.