Abstract
Highly clumped nuclei clusters captured in fluorescence in situ hybridizationmicroscopy images are common histology entities under investigations in a widespectrum of tissue-related biomedical investigations. Due to their large scalein presence, computer based image analysis is used to facilitate such analysiswith improved analysis efficiency and reproducibility. To ensure the quality ofdownstream biomedical analyses, it is essential to segment clustered nucleiwith high quality. However, this presents a technical challenge commonlyencountered in a large number of biomedical research, as nuclei are oftenoverlapped due to a high cell density. In this paper, we propose ansegmentation algorithm that identifies point pair connection candidates andevaluates adjacent point connections with a formulated ellipse fitting qualityindicator. After connection relationships are determined, we recover theresulting dividing paths by following points with specific eigenvalues fromHessian in a constrained searching space. We validate our algorithm with 560image patches from two classes of tumor regions of seven brain tumor patients.Both qualitative and quantitative experimental results suggest that ouralgorithm is promising for dividing overlapped nuclei in fluorescence in situhybridization microscopy images widely used in various biomedical research.