Lung cancer is the leading cause of cancer mortality in the US, responsiblefor more deaths than breast, prostate, colon and pancreas cancer combined.Recently, it has been demonstrated that screening those at high-risk for lungcancer low-dose computed tomography (CT) of the chest can significantly reducethis death rate. The process of evaluating a chest CT scan involves theidentification of nodules that are contained within a scan as well as theevaluation of the likelihood that a nodule is malignant based on its imagingcharacteristics. This has motivated researchers to develop image analysisresearch tools, such as nodule detectors and nodule classifiers that can assistradiologists to make accurate assessments of the patient cancer risk. In this work, we propose a two-stage framework that can assess the lungcancer risk associated with a low-dose chest CT scan. At the first stage, ourframework employs a nodule detector; while in the second stage, we use both theimage area around the nodules and nodule features as inputs to a neural networkthat estimates the malignancy risk of the whole CT scan. The proposed approach:(a) has better performance than the PanCan Risk Model, a widely accepted methodfor cancer malignancy assessment, achieving around 7% better Area Under Curvescore in two independent datasets we have employed; (b) has comparableperformance to radiologists in estimating cancer risk at patient level; (c)employs a novel multi-instance weakly-labeled approach to train the deeplearning network that requires confirmed cancer diagnosis only at the patientlevel (not at the nodule level); and (d) employs a large number of lung CTscans (more than 8000) from heterogeneous data sources (NLST, LHMC, and Kagglecompetition data) to validate and compare model performance. AUC scores for ourmodel, evaluated against confirmed cancer diagnosis, range between 82% to 90%.