Big Data/Machine Learning/AI
Development and Real-World Evaluation of a ClinicianTrained Model for Managing Patient Portal Messages Muhammad Zia ul Haq* Muhammad Zia ul Haq Blake J. Anderson Yuanda Zhu Andrew Hornback Alison D. Cowan Bradley Gallaher Arash Harzand
BACKGROUND: While patient-portal messages in the electronic health record (EHR) provide convenience and enhance patient and provider communications, their rapidly increasing volume reveals challenges in clinical practice, including delays in timely response, staff shortages, and even provider burnout. We evaluated the impact of a natural language processing (NLP) algorithm for intelligent message routing in clinical practice.
METHODS: We developed an NLP model to accurately label and route inbound patient messages using a pretrained classifier that was fine-tuned using clinician feedback. The model was prospectively deployed in an outpatient clinic environment for real-world validation. A parallel group of unrouted messages was generated for comparative analysis. The primary endpoints used to assess model performance included time to first message interaction, time to conversation resolution, and the total number of message interactions by healthcare staff, compared with the control group. Secondary endpoints were the precision, recall, F1 score (a measure of positive predictive value and sensitivity), and accuracy for correct message classification.
RESULTS: The model prospectively labeled and routed 469 unique conversations over 14 days. Compared to a control group of 402 unrouted conversations from the same period, staff in the routed message group used less time to initially address a new patient message (difference in medians, -1 hour; 95% CI 1.42 to -0.5; P<0.001) and to complete a conversation (difference in medians, -22.5 hours; 95% CI 36.3 to -17.7; P<0.001); routed group staff also had a significantly fewer number of total message interactions (difference of medians, -2.0 interactions; 95% CI –2.9 to –1.4; P<0.001). The model demonstrated high precision (>97.6%), recall (>95%), and F1 scores (>96.5%) for accurate prediction of all five message classes, with a total accuracy of 97.8%.
CONCLUSIONS: Real-time message routing using advanced NLP was associated with significantly reduced message response and resolution times and fewer overall message interactions among clinic staff.
