Identifying Signs and Symptoms of Urinary Tract Infection from Emergency Department Clinical Notes Using Large Language Models
Authors: Mark Iscoe, Vimig Socrates, Aidan Gilson, Ling Chi, Huan Li, Thomas Huang, Thomas Kearns, Rachelle Perkins, Laura Khandjian, R Andrew Taylor
Published in: Academic Emergency Medicine, 2024 April 4
Conclusions:
This study highlights the effectiveness of LLMs, particularly transformer-based models like the Clinical Longformer, in extracting symptoms of UTI from unstructured ED clinical notes. These models demonstrated high accuracy in identifying the presence or absence of UTI symptoms, although performance varied for individual symptoms. This work underscores the potential of NLP tools in enhancing EHR-based research, surveillance, and clinical decision support by efficiently processing unstructured clinical documentation.
Methods:
- Reviewed ED clinician notes for patients aged ≥18 who underwent urinalysis in a northeastern U.S. health system from June 2013 to August 2021.
- Annotated 1250 random ED notes for 17 UTI symptoms, training two LLMs for named entity recognition: a convolutional neural network-based model (SpaCy) and a transformer-based model (Clinical Longformer) designed for longer documents.
- Models were trained on 1000 notes and evaluated on a separate set of 250 notes, comparing their precision, recall, and F1 measure in identifying UTI symptoms at the note level.
Results:
- Identified 8135 entities across 1250 notes, with 83.6% of notes containing at least one UTI symptom.
- Weighted F1 measure for symptom identification at the note level was 0.84 for SpaCy and 0.88 for Longformer.
- The F1 measure for detecting any UTI symptom in a note was 0.96 for SpaCy (232/250 correctly classified) and 0.98 for Longformer (240/250 correctly classified).