The MSEP pipeline classifies sentences in clinical documents as present, absent, or unknown regarding a patient’s medical condition. Available as a Python package (repository link in Multimedia Appendix 1), the framework includes configurable parameters that allow its components to be adapted to local research needs and institutional constraints. In the following sections, we first describe the general design of the pipeline and then detail its implementation within our institutional setting.

Ethics approval for data reuse was obtained from the Commission Nationale de l'Informatique et des Libertés (agreement number 2206739). This study was conducted on pseudonymized secondary clinical data provided by the Clinical Data Center of Rennes Hospital. In accordance with French regulations and the guidelines of the Commission Nationale de l'Informatique et des Libertés. Patients were informed of the potential use of their data for research purposes through the hospital's standard information notice. All data were pseudonymized by the Clinical Data Center before being transmitted to the research team. No identifying information was accessible at any stage of the analysis. No compensation was applicable or provided to participants.

The annotated datasets become the input for training or fine-tuning neural network models for medical status extraction. Since the pipeline is designed to extract the status of one medical condition at a time, each corresponding model is trained in a multiclass classification setting, where the possible status labels (eg, present, absent, and unknown) are mutually exclusive. Model stability is assessed through cross-validation by dividing annotated data into several groups (C1), using one group for validation (C3) and the remainder for training (C2) in each session. The number of cross-validation sessions equals the number of groups of data. The data grouping strategy for cross-validation should align with study objectives; for example, testing model stability across different document types requires data partitioning by document type. The extractor’s performance metrics across multiple validation sets are assessed and compared with extractors based on other approaches, if available, to decide whether it has stable and state-of-the-art performance (C4). The flexibility of the pipeline configuration allows for easy benchmarking of different models (such as BERT-based models, rule-based systems, or LLM prompts), facilitating comprehensive performance comparison with minimal additional setup.

If the assessment result is positive, the pipeline ends, and the output is a trained medical status extraction model with satisfactory performance.

If, according to the assessment step (C4), the extractor does not achieve satisfactory performance (eg, a macro F-score below 90%), the pipeline allows for iterative refinement. The definition of “satisfactory” performance may vary depending on the objectives and standards of different research projects.

The iterative process involves both quantitative and qualitative improvements. Quantitatively, additional annotated sentences are incrementally introduced to expand the training and validation datasets. The number of added samples per iteration is not fixed and depends on multiple factors, including the time allocated for manual annotation, the number and availability of annotators, and computing resources.

Qualitatively, the pipeline may be refined by improving preannotation rules, updating annotation guidelines based on annotator feedback, or replacing rules with more accurate extractors from previous iterations for preannotation. The first iteration serves as an exploratory phase to identify bottlenecks, such as insufficient data for certain labels, weak vocabulary coverage in preannotation rules, or annotation inconsistencies, allowing for targeted improvements in subsequent iterations.

We conducted 2 iterations of the pipeline in our institutional setting to create and refine the extractors, after which most targeted medical conditions reached satisfactory performance (average macro F-score>0.94) for our use case. This section presents the results of these iterations, describing implementation paths, dataset characteristics, cross-validation results, and comparisons between our extractors and alternative approaches.

Figure 2 details the implementation path and time consumption for each medical condition across both iterations.

Most medical conditions were preannotated using rule-based matching, except smoking status, which was preannotated in the second iteration using a CamemBERT-based extractor fine-tuned from the first, which outperformed the rules. Heart failure annotation during the second iteration initially yielded insufficient interannotator agreement (57%), necessitating disagreement analysis and correcting annotated data. The analysis revealed that the inconsistencies arose from concept confusion among annotators. A list of ambiguous terms was identified and clarified with a medical expert, incorporated into updated annotation guidelines, and used to automatically extract sentences that need revision from all annotations. These sentences were then reviewed and corrected by the annotators based on the updated guidelines.

Time requirements varied by medical condition and process step. We focused on the 3 most time-intensive steps: preannotation (D1), manual annotation (M1), and extractor training during cross-validation (C2). Preannotation (D1) speed ranged from 0.02 to 0.17 seconds per sentence, with COPD and diabetes being fastest (0.02 s/sentence) and smoking and family history of cancer slowest (0.12‐0.17 s/sentence). Manual annotation (M1) required 1.2‐2.9 seconds per sentence when considering all status labels, with diabetes and family history of cancer being the most efficient (1.2 s/sentence) and smoking being the most time-intensive (2.9 s/sentence). After removing the speed bias introduced by quickly dismissed sentences labeled as unknown, annotation speed ranged from 2.23 to 4.25 s/sentence, with hypertension being the fastest and family history of cancer the slowest, indicating that annotation speed is partly driven by sentence complexity and semantic load. Cross-validation training (C2) averaged 5‐22 min per session (of cross-validation), with COPD training being fastest (5 m/session) and hypertension slowest (22 m/session). Final extractor inference speed ranged from 0.09 to 0.12 s per sentence. Total implementation time across both iterations (Ttl) ranged from 8 hours (diabetes) to 30 hours (smoking). The time consumption and speed of all automatic steps (D1, C2, and inference of extractors) are calculated on a GPU of NVIDIA A100 40 GB graphics card.

The first iteration involved quickly preannotating 9997 medical documents and selecting 991 sentences from them for manual annotation. In order to improve extractors’ performance by training them with more medical status samples, the second iteration preannotated an additional 18,994 documents, yielding 11,128 sentences for manual annotation. Table 2 shows the distribution of medical status in sentences annotated during each iteration. The training and validation dataset for the 2nd iteration of the pipeline is the combination of all sentences annotated during both iterations (12,119 sentences in total).

Sample sparsity was evident for certain status classifications. During the first iteration, most conditions had fewer than 300 samples per status, with absence samples typically fewer than presence samples. No samples were found for the absence of COPD and family history of cancer. We then increased preannotated documents for the second iteration, which successfully increased sample counts for most status classifications, though samples remained limited due to the absence of COPD and both the presence and absence of family history of cancer.

Among the 12,119 sentences, approximately 10% (n=1200) were randomly selected from the medical documents without preannotation, yielding an unbiased subset of 1200 sentences (unfiltered sentences [US]). We compared the distribution of medical statuses in this subset with that of the remaining 10,919 sentences (preannotated filtered sentences [PS]) selected through preannotation to assess the extent of bias introduced by this data qualification process (D1, D2).

Table 3 shows the result of this comparison. For each medical condition, we calculated the proportion for each of its statuses within US and PS, which is defined as:

where c() denotes the total number of samples of one or more statuses in the set. Since the purpose of the data qualification is specifically to reduce the proportion of sentences that do not express any status information (ie, the class unknown), this category was excluded from the computation to avoid inflating the proportions of relevant statuses (ie, present, absent, and former) in the PS and obscuring the assessment of distributional changes. Instead, we report the prevalence of relevant statuses in US and PS to quantify how effectively steps D1+D2 have increased relevant samples’ density in PS by filtering out irrelevant sentences (class unknown). This measurement is defined as:

Across most medical conditions, the distribution of status labels remained stable after the data filtering steps. The proportions of present status for smoking, diabetes, hypertension, and COPD, as well as the absent status for COPD and hypertension, showed only minor deviations between US and PS. A modest shift was observed for the former and absent statuses of smoking, whereas heart failure exhibited a more substantial change in the present status (from 33.3%, 1/3 in US to 47.1%, 228/484 in PS). Larger apparent shifts for rare categories, such as family history of cancer or the absent status of hypertension, reflect the extremely small number (or absence) of these samples in US, making baseline proportions unstable and limiting interpretability. Overall, despite shifts in certain conditions, the relative ordering of status frequencies remained consistent, suggesting preservation of the dominant patterns. Finally, the prevalence values show that the filtering steps substantially increased the density of relevant samples. For most conditions, the proportion of samples in PS was more than 10 times higher than in US (eg, from 0.75%, 9/1200 to 13.45%, 1469/10,919 for smoking, and from 0.25%, 3/1200 to 5.49%, 599/10,919 for diabetes), demonstrating the effectiveness of steps D1+D2 in removing irrelevant sentences while introducing only minor distributional bias.

As a reminder, each extractor was trained in a multiclass classification setting to predict the status of a single medical condition. Evaluation metrics include precision, recall, specificity, and F-score for each individual status, as well as balanced accuracy and macro F-score aggregated across all statuses. Table 4 presents the performance metrics for the best-performing extractor of each medical condition after the second iteration. Extractors for diabetes and heart failure achieved excellent overall performance (macro F-scores of 99% and 96%, respectively). The COPD extractor achieved a macro F-score of 99%, although evaluation was limited to the present and unknown classes due to the lack of absent samples. The diabetes, hypertension, and COPD extractors exceeded 98% F-score for presence detection. The smoking extractor achieved balanced performance across status classifications (93%‐94% F-scores), while the hypertension extractor showed imbalanced performance between presence (98% F-score) and absence (86% F-score). The extractor for family history of cancer underperformed with only 80% F-score for presence detection.

Table 5 presents the results of the 3-fold cross-validation from both iterations. For each medical condition, we evaluated the stability of the extractors by reporting the mean and SD of the F-score for each status category, as well as for the overall macro F-score, across the cross-validation folds. All extractors demonstrated improved performance after the second iteration. After the first iteration, only the diabetes extractor achieved a macro F-score exceeding 90%, while by the second iteration, most condition extractors surpassed this threshold. Hypertension, heart failure, and COPD extractors showed the most substantial improvement (>30%).

Despite these improvements, the family history of cancer extractor achieved only 73% macro F-score with 17% instability, primarily due to poor performance in detecting presence (55% F-score, 37% instability). The hypertension extractor also showed suboptimal performance for absence detection (85% F-score, 1% instability). Due to insufficient samples, absence classification for COPD and family history of cancer could not be trained or evaluated.

To go one step deeper into the assessment of the extractor performances, a post hoc analysis according to document type was conducted. Table 6 reports the average F-scores with SD, obtained during three-fold cross-validation during the last iteration of the pipeline, for each medical status across the major clinical document types represented in the dataset.

For 72 status–document-type combinations, no sample was retrieved in validation sets, and performance cannot be discussed (marked with an “-”). For the remaining 115 status document-type combinations, we can divide results into 2 categories.

Those with comparable performance (less than 5% decrease) or improvement on specific documents (109/115, 94.8%) and those where a decrease is observed (6/115, 5.2%). In the first category, 79 instances show a score of 1.0 +/– N/A, indicating that only one-fold contains content for evaluation. This perfect performance also suggests that few samples were available for evaluation. In the second category, 2 instances were only evaluated in one fold, and extractors missed the status (ie, for hypertension presence and heart failure presence in the nurse report). For the 4 others (smoking presence and diabetes indifference in paramedical report, absence of heart failure in daily note, and presence of COPD in administration note), a decrease is observed, but performance remains acceptable (>80%). Examination of false positives and false negatives in these 4 cases suggests that most errors stem from the misleading narrative style of clinical notes. For example, in a paramedical report, “OH chronique sevré tabac syndrome dépressif” (Chronic alcohol use weaned tobacco use depressive disorder) was misclassified as former smoking because the boundary between alcohol withdrawal and smoking was unclear. Another sentence, “des atcd cardio,HTA,DT2non traité” (A history of cardiac disease, untreated hypertension, and type 2 diabetes) was misclassified as absence of diabetes because the negation “non” was attached to the diabetes “DT2” rather than to the treatment “traité.” The extractor also produces some persistent errors due to the medical complexity of the content, which is challenging even for specialist reviewers; for instance, a sentence in a daily note was misclassified as absent of heart failure because it contains this expression “avec une fraction d'éjection ventriculaire conservée” (with a preserved ventricular ejection fraction), which does not exclude entirely the possibility of having a heart failure. In conclusion, although the representation of entities is very sparse, we observe overall good robustness of the extractors depending on the type of document encountered. To go further, the number of annotated sentences would need to be increased.

Figure 3 presents a comparison of the extractors developed using CamemBERT fine-tuning, rule-based methods, and LLM prompting (Mixtral-8×7B-v0.1). As a reminder, all extractors were evaluated on the same validation sets used for the 3-fold cross-validation of CamemBERT-based extractors. For conditions with ample annotated examples, such as smoking and heart failure, CamemBERT-based extractors achieved the highest performance (also confirmed by paired t test analysis; Multimedia Appendix 3). In contrast, for conditions with limited training data, rule-based extractors performed comparably to or better than CamemBERT-based models, reflecting their robustness in low-data scenarios. This was particularly evident for family history of cancer, where CamemBERT-based extractors achieved an average macro F-score of 0.73 with a cross-validation SD of 17%, while the rule-based approach reached an average macro F-score of 0.94 with a substantially lower SD of 7.9%.

Compared with CamemBERT-based and rule-based extractors, LLM-prompted extractors showed intermediate performance overall but were less efficient in our setting. Using Mixtral-8×7B-v0.1, sentence-level extraction required 12‐24 seconds per sentence and 35 GB of GPU memory, which is slower and more resource-intensive than CamemBERT (0.09‐0.12 s per sentence; 2 GB GPU).

These findings indicate that CamemBERT-based and rule-based methods are the most suitable approaches for our institutional resource constraints, and that no single extraction approach is universally optimal across all clinical scenarios.

This section examines the advantages and limitations of the MSEP pipeline within our institutional setting and explores how its design can support local development of clinical NLP extractors.

Our study has several limitations worth noting. First, due to the availability of medical experts, we did not conduct a direct comparison between sentence classification and NER. Instead, we relied on empirical observations and general consensus to suggest that sentence classification offers greater annotation efficiency. While the improvement in annotation speed appears supported, the impact on annotation quality remains to be verified.

Second, clinical document type is a factor that can influence the interpretation of medical status occurrence in text, which was not fully addressed in our study. For instance, a mention of “hypertension” in a medical report, especially within the “Past Medical History” section, typically refers to a chronic condition. In contrast, if hypertension is noted in daily progress notes, it is more likely to reflect an acute episode under current observation or management. Therefore, when applying MSEP to clinical documents from a CDW, it may be necessary to incorporate a document selection step during data collection (P1), not only to include relevant document types, but also to focus on specific sections within documents, such as the “Past Medical History” section, when these are more likely to align with the study objectives.

Third, the construction and usage of the datasets are subject to several sources of bias. Preannotation and filtering were used to increase the density of informative samples in the dataset, but mislabeling by preannotation rules could have led to the exclusion of relevant sentences (eg, those labeled as “unknown”). To assess this risk, 10% of our dataset consisted of nonpreannotated sentences, which were used to evaluate potential bias introduced by the preannotation rules. While most conditions showed limited distortion, a notable shift was observed for the present status of heart failure, underscoring that preannotation can influence class distributions. A further concern is the risk of circular bias, which may be induced by using sentences initially selected by rules to evaluate those same rule-based extractors. To limit this risk, we constructed the annotated corpus by combining the preannotations from all descriptors rather than relying on a single rule-based extractor per condition, resulting in a relatively small proportion of validation sentences originating from the same rule set (eg, 3.4%‐3.6% for COPD). Evaluating extractors is a complex task in such large datasets. Ideally, most of the dataset would need to be annotated to determine actual performance. However, this task requires too much annotation time for most noninformative sentences. Our approach to selection by multiple preannotations allows us to limit this time while verifying that little bias appears to be induced (eg, confirmed using sentences without preannotation). Finally, although identical sentences were deduplicated before cross-validation, dataset partitioning was not stratified by Patient_ID. As a result, sentences about the same patient could appear across validation folds, introducing a potential risk of data leakage. The proportion of validation sentences with patient overlap ranged from 3% to 8%. To assess the impact of this overlap, we revalidated the models after removing these sentences from the validation sets. Performance changes were minimal (less than 1% of drop for macro F-score of each cross-validation fold), suggesting that patient-level overlap did not artificially inflate extractor performance. Moving forward, these extractors should be used on other datasets while keeping in mind their potential limitations.

Fourth, the evaluation of certain medical statuses was limited by their very low prevalence in the corpus. For COPD and family history of cancer, the Absent status was represented by only a single instance, preventing meaningful assessment of the extractor’s ability to distinguish absent from unknown. Consequently, both extractors effectively functioned as binary classifiers (present vs unknown). This limitation reflects typical documentation practices: the absence of a specific condition (eg, COPD) is rarely stated explicitly, as doing so would require exhaustively listing all conditions that are not present; the issue is somewhat different for family history of cancer, where the absence of this condition is generally more informative (“no family history of cancer”) and therefore more likely to be mentioned, although such mentions remained infrequent in our dataset.

Finally, decoder-only LLMs were not fully used due to initial infrastructure limitations. Our computing environment, a secure server at the University Hospital of Rennes with 112 Intel(R) Xeon(R) Gold 6258R CPU cores and limited access to an NVIDIA A100 40 GB GPU, was insufficient for efficient, prompt-based inference (12‐24 s/sentence when performing medical status extraction) or fine-tuning (requiring ≥64 GB VRAM).

Beyond its current implementation, the pipeline can be extended to broader medical information extraction tasks. Ongoing work includes detecting additional statuses relevant to bladder cancer [3] and extracting finer-grained information such as smoking duration and cigarette type. By first classifying sentences containing smoking mentions, the pipeline narrows the scope for subsequent extractions, improving efficiency.

Future studies will also address current limitations. First, a direct comparison of annotation speed and quality between sentence classification and NER will provide stronger evidence for our methodological choice. Second, it is well known that medical documents contain information, structures, and sometimes vocabulary that are unique to them. For our specific study, we have demonstrated a certain robustness of extractors to these phenomena. To reinforce this finding, more instances will need to be annotated, bearing in mind that we are working in a very sparse context. Third, synthetic datasets will be explored to mitigate preannotation bias, balance category distributions, and reduce reliance on sensitive records. They may also improve extractor performance on medical conditions with scarce samples (eg, family history of cancer). Prior studies have investigated GANs for generating synthetic structured EHRs [40]; while ensuring both utility and confidentiality remain challenging, synthetic data offer a promising path forward for clinical NLP.

Finally, the Ollama framework [65] will be used for a better exploration of LLM prompt potential. This framework enables accelerated local inference, improving medical status annotation speed with Mixtral-8x7B-v0.1 to 0.2 seconds per sentence. Although LLM prompting has not yet reached the same level of performance as CamemBERT fine-tuning for certain conditions, further exploration of prompt design and fine-tuning strategies may be pursued as infrastructure improves. However, it is important to keep in mind the computational and ecological cost of such a strategy when rule-based and CamemBERT-based approaches can provide satisfactory results.

Together, these perspectives aim to strengthen the robustness, reproducibility, and applicability of the pipeline across diverse clinical research settings.

We introduced MSEP, a modular and hybrid methodological framework for extracting medical status information from unstructured clinical text using sentence classification. In our institutional case study, the pipeline substantially reduced annotation effort: sentence-level labeling with preannotation required only 1.2‐2.9 seconds per sentence. MSEP successfully supported the implementation of 3 extraction approaches: rule-based methods, fine-tuned CamemBERT model, and LLM prompting, each suited to different data and resource scenarios. Fine-tuned CamemBERT model achieved high performance for conditions with sufficient training examples (macro F-score>94%), whereas rule-based methods provided more stable results for sparsely represented conditions. Together, these findings highlight MSEP’s value as a research tool that accelerates local dataset creation and enables flexible deployment of extraction systems. Importantly, all components of the pipeline are distributed as a ready-to-use Python package.