We propose a general framework for searching surrogate losses for mainstreamsemantic segmentation metrics. This is in contrast to existing loss functionsmanually designed for individual metrics. The searched surrogate losses cangeneralize well to other datasets and networks. Extensive experiments on PASCALVOC and Cityscapes demonstrate the effectiveness of our approach. Code shall bereleased.