Performance of Natural Language Processing for Information Extraction From Electronic Health Records Within Cancer: Systematic Review

Introduction

Electronic health records (EHRs) are increasingly being adopted by health care providers worldwide, as they offer numerous benefits [1]. This has led to an increase in the quantity of data stored in EHRs, consisting of both structured and unstructured data (eg, text, images, and time series). Unstructured textual data from discharge summaries, radiology reports, clinical notes, and patient histories provide valuable information about patients that may not be captured by structured data alone [2]. The extraction of clinical parameters from unstructured textual data, also known as information extraction (IE), has proven to be valuable in health care, such as in clinical research (eg, epidemiology) and decision support systems [3,4].

However, working with unstructured textual data presents several challenges to health care providers and researchers. The volume of free text makes manual extraction and analysis time-consuming and resource-heavy, thereby limiting their utility and requiring automated solutions. Moreover, the lack of standardization and consistency in formatting and terminology makes it difficult to accurately identify and extract the relevant information in an automated manner. Furthermore, free text is prone to spelling errors, resulting in inaccurate or harder-to-find patient information for methods that rely on keyword extraction or other exact-match search techniques.

Natural language processing (NLP) techniques are well suited for extracting information from free text because of their ability to process, comprehend, and generate human language in a manner that allows for automatic extraction of structured information from free text. In recent years, NLP has gained considerable attention, with numerous studies exploring the effectiveness of various NLP techniques, notably for identifying and extracting cancer-related entities, such as smoking history [2], toxicities [5], and Gleason scores [6], which are only recorded as free text in clinical notes. These techniques are known as named entity recognition, or more generally, IE [7-9].

A variety of techniques and pipelines have been developed for IE from medical free texts, ranging from simple rule-based solutions to advanced machine learning approaches. Rule-based solutions allow domain experts to define a set of linguistic rules and patterns to be implemented to identify and extract relevant information from clinical notes and medical free-form texts. Studies have shown that rule-based approaches can outperform machine-learning models [10-13]. However, rule-based approaches are custom-made for specific datasets and use cases, require manual specification of rules by medical experts, and are therefore difficult to generalize [3]. Moreover, machine learning models allow for multiple methodologies and applications for different IE problems, solved by training specific traditional machine learning models such as support vector machines, decision trees, and neural networks (NNs). Recently, bidirectional transformers (BTs) such as large language models (LLM) have been identified as a possible tool for IE because of their strengths in pattern recognition, text summarization, and generation [14]. LLMs allow for pretraining on large text corpora, which enables them to learn linguistic patterns applicable to different IE tasks. Furthermore, LLMs show promising results for specific tasks because of the domain-specific fine-tuning of pretrained models [15].

Over the years, numerous models have been developed for various IE tasks, but their relative performance across different datasets remains largely unknown. Both rule-based and machine learning approaches often exhibit limited generalizability between tasks, and their results are frequently inconsistent when applied to different datasets. This inconsistency highlights the need for studies that investigate and compare the relative performance of these models.

To the best of our knowledge, no review has been conducted that summarizes the differences in the performance of various types of NLP models for IE within the context of cancer. This review provides an overview of the various NLP methods used for IE and compares them in terms of their performance.

Methods

This systematic review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.

We searched 3 databases—PubMed, Scopus, and Web of Science—for relevant literature published between January 1, 2014, and April 19, 2024 (Multimedia Appendix 1). The following search criteria were used for the titles and abstracts:

(“information extraction” OR “natural language processing” OR “nlp”) AND (EHR OR notes OR reports) AND (cancer OR tumor OR oncology)

The inclusion criterion for articles was the application of 2 or more NLP models to extract identical cancer-related entities from the same unstructured medical text in EHR. Articles were excluded during title and abstract screening if they were as follows: (1) reviews; (2) articles that do not compare 2 or more NLP models; (3) articles whose purpose is not to extract information from free text from EHR; and (4) articles whose purpose is not to develop an NLP cancer-related application.

For further exclusion during the full-text screening, articles were excluded if they were defined as follows: (1) abstract only; (2) text classification without cancer entity extraction; (3) results within the article were not compatible; (4) no NLP application development; (5) not related to cancer IE from EHR; and (6) no comparison with other NLP methods within the article.

Using the exclusion criteria, one author (SCD) performed 2 rounds of article selection: title and abstract screening, followed by a full-text review. A second reviewer (CV) was consulted for unclear cases during the screening.

Data from each of the included articles were extracted by 2 authors (SCD and CV). Both authors independently categorized the NLP models and extracted their performance metrics. Any discrepancies in categorization were resolved through consensus guided by consideration of the primary architectural components of the model. Each model was categorized into the following groups: rule-based, traditional machine learning (ML), conditional random field (CRF)–based, NN, and BT.

The rule-based category includes IE models that use regular expressions (Regex), keywords, and dictionary matching. The CRF-based category includes linear CRF, except bidirectional long short-term memory-CRF, which is in the NN category. The NN category includes NNs, except for BTs, that belong to the BT category. Ensemble models are categorized as the most advanced part of the ensemble. For example, a rule-based model combined with a BT is categorized as a BT model (see Table 1). For articles that included both strict and relaxed keyword matching, the strict F₁-scores were extracted as the performance metric. For articles presenting both macro- and micro-averaged F₁-scores, macro-averaged F₁-scores were extracted.

To calculate the performance differences for all categories across the included articles, the following steps were executed for all categories.

The best-performing model for each category within each article was selected. The best-performing model within category c for article a is given by ${max}_{c,a}$:

${max}_{c,a}=\max (P_{{c,m}^1},P_{{c,m}^2},\dots ,P_{c,m^n})$

where P is the F₁ performance score of method m within category c. n is the number of methods within category c.

Having the best-performing categories within an article allows for calculation of the category difference for each combination of categories. Category differences for categories c¹ and c² within article a are given as ${category\_diff}_{c^1,c^2,a}$:

${category\_diff}_{c^1,c^2,a}={max}_{c^1,a}-{max}_{c^2,a}$

where ${max}_{c,a}$ is the best performing model of category c in article a.

All performance differences of the same combination of categories were averaged across all the articles. The average of the category difference for all articles with combination c¹ and c² is given as ${performance\_difference}_{c1,c2}$ :

${\displaystyle performance\mathrm{\_}differenc{e}_{c1,c2}=\frac{category\mathrm{\_}dif{f}_{c^1,c^2,a^1}+category\mathrm{\_}dif{f}_{c^1,c^2,a^2}+\cdots +category\mathrm{\_}dif{f}_{c^1,c^2,a^n}}{n}}$

where ${category\_diff}_{c^1,c^2,a}$ is the category difference between c¹ and c² in article a. n is the number of articles with a specific category combination.

Statistical significance between categories for each category combination was determined using a t test (P<.05).

Results

Overview

The article selection process is detailed in a PRISMA flowchart, shown in Figure 1. A total of 2032 articles were identified through searches in Web of Science, Scopus, and PubMed.

‎

In total, 33 articles were included in this review. The articles were published between 2018 and 2024. They compared at least 2 NLP models for cancer-related IE from unstructured medical texts in EHRs. The articles contained a total of 220 implementations of NLP models. Selecting only the best-performing models within each category of each article summarizes 74 implementations.

Models

We categorized each NLP model as rule-based, CRF-based, BT, NN, and ML. Table 1 shows how each model was categorized and the articles in which the categories are contained.

Table 1 shows the categorization of the models, which articles contain the specific categories, and the total number of implemented models within each category.

The most frequently used category was NN, with 25 occurrences, followed by BT and ML with 16 and 11 occurrences, respectively. The most frequently implemented category was NN, with 83 implementations. The distribution of unique categorizations per year shows the variety of models that have been used throughout the years (see Figure 2). Notably, the percentage of articles on the implementation of BTs has increased over the years.

‎

Performance

The performance varied a lot according to specific use cases. Inspecting the best-performing models of the specific articles shows that a total of 5 rule-based models performed best in their articles, with F₁-scores in the range of 0.73-0.887 (see Table 2). ML did not perform best in any article despite being compared in 14 articles with a total of 39 different model implementations, and neither did CRF-based. In total, 14 articles showed that NN performed the best, with F₁-scores ranging from 0.3539 to 0.972. BT performed the best in 13 articles, with F₁-scores ranging from 0.6023 to 0.97. Looking at the raw F₁-scores, more advanced models outperformed less advanced ones.

Table 2 shows each article and the number of models in each category within the article. Parentheses show the best F₁-score for each category. The best F₁-score for each article is marked by a footnote.

Some variations between the average F₁-score performance differences were observed (see Figure 3).

Our results show that more advanced models outperform less advanced ones. The largest difference between the category performance F₁-scores was observed between the BT category and the rule-based category. BT models were compared with rule-based models in 4 studies, yielding an average performance difference of 0.2335 in terms of F₁-score. BT was the best-performing category. NN outperformed CRF-based, ML, and rule-based models, while CRF-based outperformed rule-based models, and rule-based outperformed ML models. The only statistically significant difference between categories is observed when comparing rule-based and ML; see Multimedia Appendix 2 for P values.

‎

Discussion

Principal Findings

This study provides an overview of the models used for IE in cancer and their performance in terms of the F₁-score. By including only articles with 2 or more NLP models for IE, we were able to evaluate the relative performance of each NLP within categories: rule-based, CRF-based, BT, NN, and ML.

The search string for this review combined keywords for techniques (IE and NLP), data sources (EHR, notes, reports), and the domain (cancer, tumor, and oncology) using Boolean operators to limit irrelevant results. The initial yield of 2032 articles suggests a reasonable balance, considering the stringent inclusion criteria. The “AND” clauses effectively limit the search while still including the relevant articles for the screening process. Although our search strategy included articles published from 01/01/2014, no articles prior to 2018 were included in the analysis. The reason for this discrepancy is not addressed within the scope of this review, which focused on quantifying performance differences between our categories. Notably, the most frequent reason for full-text exclusion was “No comparison with other NLP methods within the article (185 articles).” Arguing for common benchmark testing of the implemented NLP models.

Without considering a dataset or specific extraction entities, our results show that BT is the best performing category, followed by NN, CRF-based, rule-based, and ML in written order. We observed an increasing number of transformer-based models developed in recent years, with promising results. Our results highlight a pivotal moment in which BTs, such as language models, are on the verge of demonstrating their full potential in IE. Although transformers [44] and BERT [45] were introduced in 2017 and 2018, respectively, our literature review includes no articles using these technologies until 2019. This delay in time may reflect the time required for these models to become integrated into clinical research workflows. Surprisingly, rule-based solutions perform better than machine learning [13,22]. One explanation could be that rule-based solutions allow for the implementation of expert knowledge. The lowest-performing articles in terms of F₁-score do not aim to show the best possible method for extraction, but rather how F₁-scores increase using hierarchical regularization when extracting ICD-O-3 codes [42]. Similarly, the study of Park et al [43] aims to show how to increase the F₁-score, using transfer learning and zero-shot string similarity, when the number of annotated pathology reports is limited.

Multiple reviews have been conducted within the scope of NLP in a clinical context with different aims. The review by Kreimeyer et al [46] aims to identify NLP systems capable of processing clinical free text and generating structured output, thereby compiling a list of NLP solutions in use. The review by Datta et al [47] defines relevant linguistic terms by organizing unstructured clinical text related to cancer into structured data using frame semantics. The review by Bilal et al [48] examines the current state-of-the-art literature on NLP applications in analyzing EHRs and clinical notes for cancer research, quantifying the number of studies for each cancer type and outlining the research challenges and future directions for NLP when analyzing EHRs and clinical notes in cancer research. However, no review has been conducted comparing the performance of NLP models for IE of cancer-related entities from clinical text, a gap relevant to clinical informatics and crucial for improving the accuracy of cancer-related data IE within EHRs. This is the first review to summarize and compare the performance of NLP models for IE of different cancer entities from unstructured text, offering insights for clinical researchers focused on leveraging EHR data for cancer care and research.

Strengths

One strength of our study was its ability to overcome the challenge of comparing low-performing models. By including only articles with 2 or more categories, we can determine the relative performance for each paper while neglecting low-performing models from papers that do not aim to beat state-of-the-art F₁-score. Our review shows how models can be categorized and how the categorizations perform compared to each other through different datasets and extraction entities. The performance differences observed in our included articles highlight the importance of selecting the appropriate NLP model for each health care application. Our categorizations allow all models to be included, even ensemble and hybrid models. Furthermore, our performance calculation uses the best-performing model for each category reported within each included article. This approach allows for the addition of multiple new categories to support the desired level of model performance granularity.

Limitations

A categorization strategy was required to categorize all models. Most models assign into well-defined and distinct categories. However, some could be assigned to multiple categories, notably bidirectional long short-term memory-CRF models. To present intelligible results, the number of categories had to be kept relatively low, neglecting model specificities. Increasing the number of categories would reduce the number of models in that category, making the results too anecdotal. Decreasing the number of categories would increase the number of times each categorization was compared, making the averaged F₁-scores less distinctive. Ideally, we would have wished for multiple studies implementing the same set of models and categorizations to avoid certain categorizations not being compared with every other category and to avoid certain combinations of categorizations occurring only once.

We selected the F₁-score as a metric for performance; precision or recall could also be used. However, extracting specific numbers from the confusion matrix can provide deeper insights. The included studies reported F₁-scores as a measure of performance. Although this is a practical method to generate 1 performance metric, its use has some limitations. In medical IE, one could argue that false negatives are worse than false positives, potentially leading to missed diagnoses or inappropriate treatment decisions, which is not considered in the F₁-score. While metrics such as AUC-ROC, precision-recall tradeoff, or specificity offer complementary insights, their calculation was limited by the inconsistent reporting of the necessary data. Furthermore, given the sensitive nature of EHR data and the need for clinical trust, future research should also prioritize evaluating the interpretability of IE models alongside traditional performance measures to allow clinicians to understand how cancer-related entities are being extracted and validated from EHR data.

Furthermore, our included studies neglected to address the handling of negation and spelling errors. Giorgia et al [49] showed that negations account for 66% of the errors. Another study stated that BERT fails completely to show a generalizable understanding of negation, raising questions about the aptitude of language models to learn this type of meaning [50]. In this study, BTs performed well; one could wish for a general approach to analyze the errors of each model instead of the general performance derived from the confusion matrix. Negation errors pose a significant challenge in EHR data and are critical in oncology, as a misidentified negated symptom or finding can alter clinical interpretation, treatment planning, and patient care.

Perspectives

The field of IE has evolved rapidly, and models, such as LLMs, have been successfully applied in the context of cancer IE, both in terms of model performance and operational efficiency [51]. LLM could allow for enhanced transferability and utility for different IE tasks on unstructured textual data. Using LLMs for IE on unstructured textual data seems feasible because of the variety of available pretrained models in different versions. Some might perform well out of the box or with minor domain-specific fine-tuning [15]. Generally, the evaluation of LLMs is challenging because of the lack of clarity regarding whether a public benchmark dataset has been used for training. However, when using data from EHRs, it is certain that they have not been used for training a public model.

Conclusions

NLP has demonstrated the ability to identify and extract cancer-related entities from unstructured medical textual data. Generally, most of the reviewed models showed excellent performance in terms of the F₁-score, and more advanced models outperformed less advanced ones. The BT category performed the best, followed by NN. The use of BTs has increased in recent years. Rule-based applications for IE remain competitive in terms of performance in this specific context.