Abstract
Objective: To develop and evaluate a scalable methodology for harmonizinginconsistent units in large-scale clinical datasets, addressing a key barrierto data interoperability. Materials and Methods: We designed a novel unit harmonization systemcombining BM25, sentence embeddings, Bayesian optimization, and a bidirectionaltransformer based binary classifier for retrieving and matching laboratory testentries. The system was evaluated using the Optum Clinformatics Datamartdataset (7.5 billion entries). We implemented a multi-stage pipeline:filtering, identification, harmonization proposal generation, automatedre-ranking, and manual validation. Performance was assessed using MeanReciprocal Rank (MRR) and other standard information retrieval metrics. Results: Our hybrid retrieval approach combining BM25 and sentence embeddings(MRR: 0.8833) significantly outperformed both lexical-only (MRR: 0.7985) andembedding-only (MRR: 0.5277) approaches. The transformer-based reranker furtherimproved performance (absolute MRR improvement: 0.10), bringing the finalsystem MRR to 0.9833. The system achieved 83.39\% precision at rank 1 and94.66\% recall at rank 5. Discussion: The hybrid architecture effectively leverages the complementarystrengths of lexical and semantic approaches. The reranker addresses caseswhere initial retrieval components make errors due to complex semanticrelationships in medical terminology. Conclusion: Our framework provides an efficient, scalable solution for unitharmonization in clinical datasets, reducing manual effort while improvingaccuracy. Once harmonized, data can be reused seamlessly in different analyses,ensuring consistency across healthcare systems and enabling more reliablemulti-institutional studies and meta-analyses.