Although neural models have achieved impressive results on several NLPbenchmarks, little is understood about the mechanisms they use to performlanguage tasks. Thus, much recent attention has been devoted to analyzing thesentence representations learned by neural encoders, through the lens of`probing' tasks. However, to what extent was the information encoded insentence representations, as discovered through a probe, actually used by themodel to perform its task? In this work, we examine this probing paradigmthrough a case study in Natural Language Inference, showing that models canlearn to encode linguistic properties even if they are not needed for the taskon which the model was trained. We further identify that pretrained wordembeddings play a considerable role in encoding these properties rather thanthe training task itself, highlighting the importance of careful controls whendesigning probing experiments. Finally, through a set of controlled synthetictasks, we demonstrate models can encode these properties considerably abovechance-level even when distributed in the data as random noise, calling intoquestion the interpretation of absolute claims on probing tasks.