Traditional NLP has long held (supervised) syntactic parsing necessary forsuccessful higher-level semantic language understanding (LU). The recent adventof end-to-end neural models, self-supervised via language modeling (LM), andtheir success on a wide range of LU tasks, however, questions this belief. Inthis work, we empirically investigate the usefulness of supervised parsing forsemantic LU in the context of LM-pretrained transformer networks. Relying onthe established fine-tuning paradigm, we first couple a pretrained transformerwith a biaffine parsing head, aiming to infuse explicit syntactic knowledgefrom Universal Dependencies treebanks into the transformer. We then fine-tunethe model for LU tasks and measure the effect of the intermediate parsingtraining (IPT) on downstream LU task performance. Results from both monolingualEnglish and zero-shot language transfer experiments (with intermediatetarget-language parsing) show that explicit formalized syntax, injected intotransformers through IPT, has very limited and inconsistent effect ondownstream LU performance. Our results, coupled with our analysis oftransformers' representation spaces before and after intermediate parsing, makea significant step towards providing answers to an essential question: how(un)availing is supervised parsing for high-level semantic natural languageunderstanding in the era of large neural models?