Anomaly detectors are often designed to catch statistical anomalies.End-users typically do not have interest in all of the detected outliers, butonly those relevant to their application. Given an existing black-boxsequential anomaly detector, this paper proposes a method to improve its userrelevancy using a small number of human feedback. As our first contribution,the method is agnostic to the detector: it only assumes access to its anomalyscores, without requirement on any additional information inside it. Inspiredby a fact that anomalies are of different types, our approach identifies thesetypes and utilizes user feedback to assign relevancy to types. This relevancyscore, as our second contribution, is used to adjust the subsequent anomalyselection process. Empirical results on synthetic and real-world datasets showthat our approach yields significant improvements on precision and recall overa range of anomaly detectors.