Studying caricature recognition is fundamentally important to understandingof face perception. However, little research has been conducted in the computervision community, largely due to the shortage of suitable datasets. In thispaper, a new caricature dataset is built, with the objective to facilitateresearch in caricature recognition. All the caricatures and face images werecollected from the Web. Compared with two existing datasets, this dataset ismuch more challenging, with a much greater number of available images, artisticstyles and larger intra-personal variations. Evaluation protocols are alsooffered together with their baseline performances on the dataset to allow faircomparisons. Besides, a framework for caricature face recognition is presentedto make a thorough analyze of the challenges of caricature recognition. Byanalyzing the challenges, the goal is to show problems that worth to be furtherinvestigated. Additionally, based on the evaluation protocols and theframework, baseline performances of various state-of-the-art algorithms areprovided. A conclusion is that there is still a large space for performanceimprovement and the analyzed problems still need further investigation.