Computer Aided Diagnosis (CAD) tools are often needed for fast and accuratedetection, characterization, and risk assessment of different tumors fromradiology images. Any improvement in robust and accurate image-based tumorcharacterization can assist in determining non-invasive cancer stage,prognosis, and personalized treatment planning as a part of precision medicine.In this study, we propose both supervised and unsupervised machine learningstrategies to improve tumor characterization. Our first approach is based onsupervised learning for which we demonstrate significant gains in deep learningalgorithms, particularly Convolutional Neural Network (CNN), by utilizingcompletely 3D approach and transfer learning to address the requirements ofvolumetric and large amount of training data, respectively. Motivated by theradiologists' interpretations of the scans, we then show how to incorporatetask dependent feature representations into a CAD system via a graphregularized sparse Multi-Task Learning (MTL) framework. In the second approach, we explore an unsupervised scheme in order to addressthe limited availability of labeled training data, a common problem in medicalimaging applications. Inspired by learning from label proportion (LLP)approaches, we propose a new algorithm, proportion-SVM, to characterize tumortypes. In this second approach, we also seek the answer to the fundamentalquestion about the goodness of "deep features" for unsupervised tumorclassification. Finally, we study the effect of unsupervised representationlearning using Generative Adversarial Networks (GAN) on classificationperformance. We evaluate our proposed approaches (both supervised andunsupervised) on two different tumor diagnosis challenges: lung and pancreaswith 1018 CT and 171 MRI scans respectively.