TY  - JOUR
AU  - Ohno, Yukiko
AU  - Aomori, Tohru
AU  - Nishiyama, Tomohiro
AU  - Kato, Riri
AU  - Fujiki, Reina
AU  - Ishikawa, Haruki
AU  - Kiyomiya, Keisuke
AU  - Isawa, Minae
AU  - Mochizuki, Mayumi
AU  - Aramaki, Eiji
AU  - Ohtani, Hisakazu
PY  - 2025
DA  - 2025/3/4
TI  - Performance Improvement of a Natural Language Processing Tool for Extracting Patient Narratives Related to Medical States From Japanese Pharmaceutical Care Records by Increasing the Amount of Training Data: Natural Language Processing Analysis and Validation Study
JO  - JMIR Med Inform
SP  - e68863
VL  - 13
KW  - natural language processing
KW  - NLP
KW  - named entity recognition
KW  - NER
KW  - deep learning
KW  - pharmaceutical care record
KW  - electronic medical record
KW  - EMR
KW  - Japanese
AB  - Background: Patients’ oral expressions serve as valuable sources of clinical information to improve pharmacotherapy. Natural language processing (NLP) is a useful approach for analyzing unstructured text data, such as patient narratives. However, few studies have focused on using NLP for narratives in the Japanese language. Objective: We aimed to develop a high-performance NLP system for extracting clinical information from patient narratives by examining the performance progression with a gradual increase in the amount of training data. Methods: We used subjective texts from the pharmaceutical care records of Keio University Hospital from April 1, 2018, to March 31, 2019, comprising 12,004 records from 6559 cases. After preprocessing, we annotated diseases and symptoms within the texts. We then trained and evaluated a deep learning model (bidirectional encoder representations from transformers combined with a conditional random field [BERT-CRF]) through 10-fold cross-validation. The annotated data were divided into 10 subsets, and the amount of training data was progressively increased over 10 steps. We also analyzed the causes of errors. Finally, we applied the developed system to the analysis of case report texts to evaluate its usability for texts from other sources. Results: The F1-score of the system improved from 0.67 to 0.82 as the amount of training data increased from 1200 to 12,004 records. The F1-score reached 0.78 with 3600 records and was largely similar thereafter. As performance improved, errors from incorrect extractions decreased significantly, which resulted in an increase in precision. For case reports, the F1-score also increased from 0.34 to 0.41 as the training dataset expanded from 1200 to 12,004 records. Performance was lower for extracting symptoms from case report texts compared with pharmaceutical care records, suggesting that this system is more specialized for analyzing subjective data from pharmaceutical care records. Conclusions: We successfully developed a high-performance system specialized in analyzing subjective data from pharmaceutical care records by training a large dataset, with near-complete saturation of system performance with about 3600 training records. This system will be useful for monitoring symptoms, offering benefits for both clinical practice and research. 
SN  - 2291-9694
UR  - https://medinform.jmir.org/2025/1/e68863
UR  - https://doi.org/10.2196/68863
UR  - http://www.ncbi.nlm.nih.gov/pubmed/40053805
DO  - 10.2196/68863
ID  - info:doi/10.2196/68863
ER  -