The earth system is exceedingly complex and often chaotic in nature, makingprediction incredibly challenging: we cannot expect to make perfect predictionsall of the time. Instead, we look for specific states of the system that leadto more predictable behavior than others, often termed "forecasts ofopportunity". When these opportunities are not present, scientists needprediction systems that are capable of saying "I don't know." We introduce anovel loss function, termed "abstention loss", that allows neural networks toidentify forecasts of opportunity for regression problems. The abstention lossworks by incorporating uncertainty in the network's prediction to identify themore confident samples and abstain (say "I don't know") on the less confidentsamples. The abstention loss is designed to determine the optimal abstentionfraction, or abstain on a user-defined fraction via a PID controller. Unlikemany methods for attaching uncertainty to neural network predictionspost-training, the abstention loss is applied during training to preferentiallylearn from the more confident samples. The abstention loss is built upon astandard computer science method. While the standard approach is itself asimple yet powerful tool for incorporating uncertainty in regression problems,we demonstrate that the abstention loss outperforms this more standard methodfor the synthetic climate use cases explored here. The implementation ofproposed loss function is straightforward in most network architecturesdesigned for regression, as it only requires modification of the output layerand loss function.