The success of deep learning has been due in no small part to theavailability of large annotated datasets. Thus, a major bottleneck in thecurrent learning pipeline is the human annotation of data, which can be quitetime consuming. For a given problem setting, we aim to circumvent this issuevia the use of an externally specified energy function appropriate for thatsetting; we call this the "Deep Energy" approach. We show how to train anetwork on an entirely unlabelled dataset using such an energy function, andapply this general technique to learn CNNs for two specific tasks: seededsegmentation and image matting. Once the network parameters have been learned,we obtain a high-quality solution in a fast feed-forward style, without theneed to repeatedly optimize the energy function for each image.