Learning template based information extraction from documents is a crucialyet difficult task. Prior template-based IE approaches assume foreknowledge ofthe domain templates; however, real-world IE do not have pre-defined schemasand it is a figure-out-as you go phenomena. To quickly bootstrap templates in areal-world setting, we need to induce template slots from documents with zeroor minimal supervision. Since the purpose of question answering intersect withthe goal of information extraction, we use automatic question generation toinduce template slots from the documents and investigate how a tiny amount of aproxy human-supervision on-the-fly (termed as InteractiveIE) can further boostthe performance. Extensive experiments on biomedical and legal documents, whereobtaining training data is expensive, reveal encouraging trends of performanceimprovement using InteractiveIE over AI-only baseline.