Sarcasm is a sophisticated speech act which commonly manifests on socialcommunities such as Twitter and Reddit. The prevalence of sarcasm on the socialweb is highly disruptive to opinion mining systems due to not only its tendencyof polarity flipping but also usage of figurative language. Sarcasm commonlymanifests with a contrastive theme either between positive-negative sentimentsor between literal-figurative scenarios. In this paper, we revisit the notionof modeling contrast in order to reason with sarcasm. More specifically, wepropose an attention-based neural model that looks in-between instead ofacross, enabling it to explicitly model contrast and incongruity. We conductextensive experiments on six benchmark datasets from Twitter, Reddit and theInternet Argument Corpus. Our proposed model not only achieves state-of-the-artperformance on all datasets but also enjoys improved interpretability.