Clinical notes contain information about patients that goes beyond structureddata like lab values and medications. However, clinical notes have beenunderused relative to structured data, because notes are high-dimensional andsparse. This work develops and evaluates representations of clinical notesusing bidirectional transformers (ClinicalBert). ClinicalBert uncovershigh-quality relationships between medical concepts as judged by humans.ClinicalBert outperforms baselines on 30-day hospital readmission predictionusing both discharge summaries and the first few days of notes in the intensivecare unit. Code and model parameters are available.