Enhancing machine capabilities to answer questions has been a topic ofconsiderable focus in recent years of NLP research. Language models likeEmbeddings from Language Models (ELMo) and Bidirectional EncoderRepresentations from Transformers (BERT)  have been very successful indeveloping general purpose language models that can be optimized for a largenumber of downstream language tasks. In this work, we focused on augmenting thepre-trained BERT language model with different output neural net architecturesand compared their performance on question answering task posed by the StanfordQuestion Answering Dataset 2.0 (SQUAD 2.0) . Additionally, we alsofine-tuned the pre-trained BERT model parameters to demonstrate itseffectiveness in adapting to specialized language tasks. Our best outputnetwork, is the contextualized CNN that performs on both the unanswerable andanswerable question answering tasks with F1 scores of 75.32 and 64.85respectively.