We develop a new Low-level, First-order Probabilistic Programming Language(LF-PPL) suited for models containing a mix of continuous, discrete, and/orpiecewise-continuous variables. The key success of this language and itscompilation scheme is in its ability to automatically distinguish parametersthe density function is discontinuous with respect to, while further providingruntime checks for boundary crossings. This enables the introduction of newinference engines that are able to exploit gradient information, whileremaining efficient for models which are not everywhere differentiable. Wedemonstrate this ability by incorporating a discontinuous Hamiltonian MonteCarlo (DHMC) inference engine that is able to deliver automated and efficientinference for non-differentiable models. Our system is backed up by amathematical formalism that ensures that any model expressed in this languagehas a density with measure zero discontinuities to maintain the validity of theinference engine.