Many current NLP systems are built from language models trained to optimizeunsupervised objectives on large amounts of raw text. Under what conditionsmight such a procedure acquire meaning? Our systematic experiments withsynthetic data reveal that, with languages where all expressions havecontext-independent denotations (i.e., languages with strong transparency),both autoregressive and masked language models successfully learn to emulatesemantic relations between expressions. However, when denotations are changedto be context-dependent with the language otherwise unmodified, this abilitydegrades. Turning to natural language, our experiments with a specificphenomenon -- referential opacity -- add to the growing body of evidence thatcurrent language models do not represent natural language semantics well. Weshow this failure relates to the context-dependent nature of natural languageform-meaning mappings.