Speaker identification systems are deployed in diverse environments, oftendifferent from the lab conditions on which they are trained and tested. In thispaper, first, we show the problem of generalization using fixed thresholdscomputed using the equal error rate metric. Secondly, we introduce a novel andgeneralizable speaker-specific thresholding technique for robust imposteridentification in unseen speaker identification. We propose a speaker-specificadaptive threshold, which can be computed using the enrollment audio samples,for identifying imposters in unseen speaker identification. Furthermore, weshow the efficacy of the proposed technique on VoxCeleb1, VCTK and the FFSVC2022 datasets, beating the baseline fixed thresholding by up to 25%. Finally,we exhibit that the proposed algorithm is also generalizable, demonstrating itsperformance on ResNet50, ECAPA-TDNN and RawNet3 speaker encoders.