Semantic matching of natural language sentences or identifying therelationship between two sentences is a core research problem underlying manynatural language tasks. Depending on whether training data is available, priorresearch has proposed both unsupervised distance-based schemes and superviseddeep learning schemes for sentence matching. However, previous approacheseither omit or fail to fully utilize the ordered, hierarchical, and flexiblestructures of language objects, as well as the interactions between them. Inthis paper, we propose Hierarchical Sentence Factorization---a technique tofactorize a sentence into a hierarchical representation, with the components ateach different scale reordered into a "predicate-argument" form. The proposedsentence factorization technique leads to the invention of: 1) a newunsupervised distance metric which calculates the semantic distance between apair of text snippets by solving a penalized optimal transport problem whilepreserving the logical relationship of words in the reordered sentences, and 2)new multi-scale deep learning models for supervised semantic training, based onfactorized sentence hierarchies. We apply our techniques to text-pairsimilarity estimation and text-pair relationship classification tasks, based onmultiple datasets such as STSbenchmark, the Microsoft Research paraphraseidentification (MSRP) dataset, the SICK dataset, etc. Extensive experimentsshow that the proposed hierarchical sentence factorization can be used tosignificantly improve the performance of existing unsupervised distance-basedmetrics as well as multiple supervised deep learning models based on theconvolutional neural network (CNN) and long short-term memory (LSTM).