In pharmacoepidemiologic research, understanding adverse event trajectories and the clinical intent underlying drug prescribing requires narrative context embedded in free-text electronic health records (EHRs) [1,2]. However, structured data alone (eg, diagnosis codes and medication orders) often fail to capture the reasons for clinical decisions or the causal relationships between events, leading to critical information gaps that limit analytical validity and interpretability [3-5]. Manual chart review remains the gold standard for addressing these gaps, but it is labor-intensive, difficult to scale to large datasets, and prone to extraction errors [6,7].

Unstructured EHR narratives are increasingly recognized as a valuable resource for enriching structured datasets with clinically meaningful context. However, sharing raw clinical text is constrained by stringent privacy requirements [2,5,6,8]. Conventional deidentification methods typically replace protected health information (PHI) with generic tags or remove it entirely. These approaches can obscure subtle yet clinically important expressions, resulting in a substantial loss of analytical utility and limiting the ability to audit and validate findings against the original narrative context [1,9]. This challenge is particularly pronounced in Japanese clinical text, which often includes mixed Japanese-English expressions, ambiguous negation, and orthographic variation, making context-preserving automated deidentification especially difficult [10,11].

This study aimed to develop a framework that simultaneously (1) extracts structured information from Japanese EHR free text using BERT (Bidirectional Encoder Representations from Transformers)–based natural language processing (NLP) and (2) generates deidentified analytic text under governance constraints that preserves sufficient context to support post hoc validation.

We implemented a multistep deidentification process that begins with fully masked text and selectively restores clinically relevant keywords while retaining selected contextual expressions relevant to downstream interpretation and essential medical terminology. This approach was designed to reduce direct identifier exposure while preserving clinically relevant contextual cues, enabling the assessment of interevent relationships and clinical reasoning that are difficult to infer from structured data alone.

Using an oncology follow-up setting, we demonstrated how the combined use of structured variables and context-preserving deidentified narratives can enhance the reliability and efficiency of pharmacoepidemiologic research.

This retrospective observational study used the DATuM IDEA database, which was developed by TOPPAN Holdings Inc using data provided by the Medical Information Management Organization of the Japan Medical Association (J-MIMO) and Integrated Clinical Care Informatics Inc (ICI). Both J-MIMO and ICI are business operators certified by the Cabinet Office under Japan’s Act on Anonymized Medical Data That Are Meant to Contribute to Research and Development in the Medical Field (Act No. 28 of 2017; commonly referred to as the Next-Generation Medical Infrastructure Act) [12], which authorizes certified operators to create and provide anonymized medical data for research and development purposes.

The DATuM IDEA database integrates routinely collected EHR data including unstructured clinical narratives (eg, discharge summaries and progress notes), with structured records such as claims, prescriptions, and procedure or surgery data from 2 oncology care hospitals in Japan that provide both inpatient and outpatient services. Hospital names are not disclosed because of data governance requirements. The study data covered the period from January 1, 2019, through June 30, 2025. Because the data were retrospectively collected, a time lag existed between data generation and availability for analysis.

Under the governance framework established by the Act, data are collected and made available for research through patient notification and an opt-out mechanism. Before researcher access, data collection, linkage, and deidentification are conducted under certified governance, and TOPPAN Holdings Inc standardizes the EHR and claims data and provides the DATuM IDEA research database for approved projects. This study used deidentified data made available for research in accordance with the applicable legal and governance framework in Japan.

This study was conducted in accordance with the Declaration of Helsinki and under Japan’s Next-Generation Medical Infrastructure Act. The research and development plan was reviewed and approved by the Processed Information Provision Review Committee of the J-MIMO (approval number: 2023‐2). Because the data were collected under a patient notification and opt-out framework and provided in deidentified form under certified governance, the requirement for individual informed consent was waived.

Identifiable source data were retained and processed within ICI. Deterministic linkage and deidentification were completed before data release. Researchers at TOPPAN accessed only deidentified outputs (including deidentified analytic text) and had no access to direct identifiers.

Within ICI, deterministic linkage was performed before deidentification at each participating hospital using facility-specific patient chart identifiers to link unstructured EHR narratives with structured records (claims, prescriptions, and procedure or surgery data). Patient identifiers were facility-specific and were not harmonized across participating facilities; therefore, the same individual appearing at different facilities, if applicable, would be represented by separate database person identifiers. Structured record linkage was available for 4346 of 4383 patients in the linked text cohort (99.2%). Unlinked records were excluded from analyses requiring structured data concordance.

Eligible patients were those with at least one recorded diagnosis within International Statistical Classification of Diseases, Tenth Revision (ICD-10) codes C00-D48 (neoplasms) during the study period. Because cancer diagnoses may have occurred before the start of the data capture period, “first recorded diagnosis” was defined as the earliest observed cancer diagnosis within the available dataset; we note that, in the database, the recorded diagnosis date could precede the period during which structured data become fully observable, which was a specification of this database rather than a general property of EHR data (see the Observation Time Definition section). The linked text cohort that served as the source text population for the deidentification framework consisted of all eligible patients with at least one linked progress note. All downstream recoverability and proxy-based concordance analyses reported in this manuscript were conducted in the downstream analyzable text cohort (Layer C), defined as patients meeting all of the following: (1) included in the linked text cohort; (2) first recorded cancer diagnosis on or after January 1, 2019; (3) ICD-10 level-3 category not corresponding to secondary or unspecified neoplasms; and (4) belonging to an ICD-10 level-3 category in which at least 50 patients met these criteria. Recoverable TNM (tumor-node-metastasis) or stage-related mentions identified within this cohort were summarized separately as the downstream recoverable subset (Layer D). Cohort sizes derived under these criteria are reported in the Study Population and Analytic Flow section. Accordingly, downstream recoverability estimates were calculated within this analyzable cohort, with Layer C used for marginal recoverability and Layer D used for conditional recoverability. These estimates should not be interpreted as performance estimates for an unselected representative corpus of all linked progress notes.

Notes were not preselected on the basis of prior chart confirmation of TNM or stage documentation; the entity extraction pipeline was applied to all linked progress notes in the eligible patient cohort. The construction of the analytic cohort is acknowledged in interpretation: Reported recoverability metrics characterize the pipeline’s behavior within this analyzable subset rather than its marginal recovery probability over an unselected corpus. For proxy-based concordance analyses (see the Evaluation section), only cancer types with at least 50 eligible patients in the analytic cohort were included, and stratified analyses were performed based on ICD-10 level-3 category.

Of the 3689 patients in Layer C, 1556 (42.2%) were classified as incident (first recorded cancer diagnosis ≥30 days after the data capture start date for their hospital), 2016 (54.6%) were classified as prevalent (first diagnosis predating the data capture start date), and 117 (3.2%) were considered unclassifiable (0‐29 d from data capture start). Among incident patients (n=1556), the proportions with at least one ATC code L record within 30, 60, and 90 days of first diagnosis were 14.8% (230/1556), 21% (327/1556), and 24.3% (378/1556), respectively—stable across window specifications. Prevalent patients showed markedly lower rates (8/2016, 0.4%; 30/2016, 1.5%; and 62/2016, 3.1%, for 30, 60, and 90 d, respectively), consistent with treatment history predating the database observation window. All-patient rates across the same time windows were 6.8% (251/3689), 10.2% (376/3689), and 12.6% (465/3689), respectively. These results are summarized in Table S4 in Multimedia Appendix 1.

A manual audit of the generated deidentified analytic text was conducted primarily to inspect for apparent residual direct identifiers. Retention of intended TNM classification and cancer stage outputs after normalization was reviewed as a secondary sanity check. The audit included a stratified sample of 400 deidentified narratives limited to cancer types with at least 50 verifiable cases, with allocation proportional to patient counts per cancer type. TNM classification and stage were manually reviewed within the audited generated outputs to assess whether the intended extracted entities were retained after normalization.

Given that the deidentified analytic text is generated to explicitly reflect entities extracted by the NLP pipeline, this audit served as a sanity check for entity retention and residual PHI inspection rather than as a formal gold standard evaluation of general NLP performance (eg, precision, recall, and F₁) on raw clinical narratives. For operational data release, a multilayer quality assurance process is planned, combining sample-based manual review with automated PHI screening using SQL-based pattern checks.

Deterministic linkage was performed within ICI before deidentification using facility-specific patient identifiers. Among the 4383 patients in the linked text cohort, structured-record linkage was available for 4346 patients (99.2%) for downstream proxy-based concordance analyses.

Two independent annotators each annotated the same document set for target clinical entities (Person 1: 861 entities; Person 2: 904 entities). Strict span-level IAA, requiring agreement on both span boundaries and entity label, achieved an F₁ of 0.593 (precision: 0.579, recall: 0.607), with Person 1 treated as the reference set and Person 2 treated as the prediction set for the reported calculation; the value was symmetric and unchanged when the roles were reversed. Cohen kappa at the entity-label level was 0.766, indicating substantial agreement. After joint reconciliation, the gold standard reference set contained 955 entities. Against this gold standard, the pretraining baseline achieved an F₁ of 0.384 (precision: 0.329, recall: 0.461; 1339 predicted entities). The fine-tuned model achieved an F₁ of 0.557 (precision: 0.552, recall: 0.561; 971 predicted entities), an absolute F₁ improvement of +0.173 (+45% relative), and 27% reduction in over-prediction. The fine-tuned model F₁ of 0.557 fell within 0.036 points of the IAA ceiling of 0.593 (Table S2 in Multimedia Appendix 1).

These results provided an initial internal source text reference evaluation of the extraction component and were distinct from the study-level recoverability analyses reported below for the governance-constrained analyzable cohort based on deidentified analytic text.

From the DATuM IDEA database, 51,876 patients had at least one recorded diagnosis. Among them, 10,214 patients had a diagnosis of neoplasms (ICD-10 C00-D48) recorded on or after January 1, 2019. Linked progress notes were available for 4383 patients (82,863 records); this linked text cohort served as the source text population for the deidentification framework. Application of the downstream processing criteria described in the Study Population section yielded a downstream analyzable text cohort (Layer C) of 3689 patients and 38,841 documents that was used for downstream recoverability and proxy-based concordance analyses. Of these, 1339 patients (Layer D, 36.3% marginal patient-level recoverability) and 11,914 documents (30.7% marginal document-level recoverability) had at least one recoverable TNM or stage mention. See Figure 1 for the updated cohort attrition diagram. Accordingly, downstream recoverability estimates were interpreted within the governance-constrained Layer C cohort; estimates reported specifically for Layer D were interpreted as conditional on the presence of at least one recoverable TNM or stage mention.

The flow diagram summarizes patients with at least one recorded diagnosis in DATuM IDEA (n=51,876), patients with neoplasms (ICD-10 C00-D48; n=10,214), patients in the linked text cohort with available linked progress notes (n=4383; n=82,863 records), and the downstream analyzable text cohort (Layer C) used for downstream recoverability and proxy-based concordance analyses (Layer C: 3689 patients and 38,841 documents; Layer D: 1339 patients and 11,914 documents). Deterministic linkage between unstructured EHR narratives and structured records was performed within ICI prior to deidentification, with structured record linkage available for 4346 of 4383 patients (99.2%). Hospital names were not disclosed.

Patient and document characteristics for 6 major ICD-10 level-3 cancer subgroups within the downstream analyzable text cohort (Layer C) are summarized in Table 2.

We evaluated TNM recoverability and assessed both internal consistency and concordance with structured data within the downstream analyzable text cohort (Layer C: 3689 patients, 38,841 documents). Within the Layer D subset (11,914 documents), the document-level recoverability rates for TNM elements were 0.788 for T, 0.787 for N, and 0.788 for M. These rates reflect the proportion of analyzed documents in which each element was recoverable and normalized within the governance-constrained analyzable cohort, corresponding to Layer C document-level marginal recoverability of 24.2% for T, N, and M elements and 23.6% for stage (see the Panel A information in Table 3 as well as Panel B information in Table 3 for M1/Stage IV concordance metrics). Internal concordance between independently extracted M1 and Stage IV mention indicators showed an overall agreement of 94.7%, a PPA of 54.8%, an NPA of approximately 99.0%, and a kappa near zero because of structural imbalance. Separately, the Jaccard coefficients for T and stage were 0.080 and 0.096, respectively, at the document level and 0.169 and 0.168, respectively, at the patient level.

In the structured data proxy analysis, Stage IV classification using systemic anticancer therapy (ATC code L) showed a sensitivity of 0.257, specificity of 0.822, and positive predictive value (PPV) of 0.700. Early-stage classification using cancer-directed procedures as a proxy for curative-intent treatment showed a sensitivity of 0.549, specificity of 0.582, and PPV of 0.031. Notably, cancer-specific PPVs for Stage IV were 0.800 (breast cancer), 0.800 (prostate cancer), 0.749 (lung cancer), 0.526 (rectal cancer), 0.456 (colon cancer), and 0.798 (gastric cancer); corresponding PPVs for early-stage classification were 0.023, 0.026, 0.008, 0.037, 0.065, and 0.035, respectively (see the Panel D information in Table 3).

We evaluated the deidentification process by reporting masking and restoration rates across ICD-10 categories to characterize the privacy-utility balance of the generated analytic texts. Deidentification metrics stratified by ICD-10 category are summarized in Table 4.

The median masking rate ranged from 9.9% (C20, rectal cancer) to 15% (C34, lung cancer). The upper quartile (Q3) masking rate was highest for prostate cancer (C61; 21.3%). Corresponding restoration rates (100 − masking rate) are provided as a descriptive index of residual visible character density.

We also conducted a targeted manual audit of the generated deidentified analytic text using a stratified sample of 400 narratives. Among audited documents containing an intended TNM or stage output, all prespecified outputs were retained after normalization, and no apparent residual direct identifiers, including personal names, initials, facility names, addresses, telephone numbers, and patient IDs, were identified in any of the 400 audited narratives. Because the deidentified analytic text explicitly reflects entities generated through the extraction pipeline, this audit should be interpreted as a sanity check for output retention and PHI inspection rather than as a formal benchmark evaluation of precision, recall, or F₁ on raw clinical narratives.

We developed a framework that provides both (1) structured variables extracted from Japanese EHR free text (eg, TNM elements and cancer stage) and (2) context-preserving deidentified analytic text to enable auditing of extracted outputs. In DATuM IDEA, 10,214 patients had a diagnosis of neoplasms (ICD-10 C00-D48), and linked progress notes were available for 4383 patients (82,863 records); this linked text cohort defined the source text population for the deidentification framework. All downstream recoverability and proxy-based concordance analyses were conducted in the downstream analyzable text cohort (Layer C: 3689 patients and 38,841 documents), defined by the criteria described in the Study Population section. Deterministic linkage was performed within ICI before deidentification, and structured-record linkage was available for 4346 of 4383 patients (99.2%) in the linked text cohort.

Our framework was designed to reduce reidentification risk while preserving narrative structure, including clinically meaningful terms, function words, and contextual cues. Previous work has addressed deidentification of Japanese EHR free text [9], whereas our framework further emphasizes the provision of deidentified analytic text that remains useful for auditing and reproducible extraction in studies using real-world data. This approach is particularly valuable in studies using real-world data, where structured data alone may not capture clinical intent, negation, uncertainty, or temporal context. Prior work has also shown that deidentification may affect downstream clinical NLP performance, underscoring the practical importance of balancing privacy protection against analytic utility rather than treating masking as a purely one-way process [10]. Providing auditable deidentified analytic text alongside structured variables offers a practical alternative to full manual chart review under privacy constraints.

Within the Layer D subset of 11,914 documents, the conditional document-level recoverability rates were 0.788 for T, 0.787 for N, and 0.788 for M. In contrast, across all Layer C documents, the marginal document-level recoverability rate was 24.2% for each of the T, N, and M elements and 23.6% for stage. The independently extracted M1 and Stage IV mention indicators showed high overall agreement, with an overall agreement of 94.7%, a PPA of 54.8%, and an NPA of approximately 99%. Cohen kappa remained near zero because of low positive prevalence, asymmetric positive rates, and a dominant concordant-negative cell. The TNM recoverability estimates should be interpreted in relation to the aforementioned Layer C and Layer D denominators and not as benchmark performance estimates from an unselected corpus of all oncology notes.

In the structured data proxy analysis, Stage IV classification assessed against systemic anticancer therapy (ATC code L) showed a PPV of 0.700, supporting the plausibility of advanced/metastatic classification when an observable treatment proxy is available. In contrast, early-stage disease (Stage I–II) assessed in relation to cancer-directed procedures as a proxy for curative-intent treatment exhibited a low PPV (0.031).

This finding should not be interpreted as definitive evidence of incorrect stage assignment by the NLP component. Rather, it likely reflects limitations of using procedure-based proxies in routinely collected structured data. In particular, (1) early-stage cancer management is not uniformly surgical (eg, endoscopic therapy, radiotherapy, or active surveillance), (2) procedures may not be observed if care is delivered outside the contributing hospitals or outside the observation window, (3) timing misalignment can occur between narrative documentation of stage and billing-based capture of procedures, and (4) stage statements in notes may sometimes refer to prior or concurrent malignancies rather than the indexed primary cancer. More broadly, the use of structured treatment data as a proxy-based concordance strategy is consistent with a line of EHR research in which information extracted from narrative records is evaluated through downstream clinical consistency and treatment-related variables rather than through a single universal gold standard [11]. Accordingly, proxy-based concordance for early-stage disease should be interpreted as concordance with an observable treatment proxy within the available data window, rather than as verification of the true underlying pathologic stage. Future work will refine proxy-based concordance analyses using composite treatment proxies—including endoscopic therapy and radiotherapy—to further assess the robustness of the classification framework.

A targeted manual audit was performed on a stratified sample of 400 deidentified narratives from cancer types with at least 50 verifiable cases. Two independent auditors reviewed the narratives, including approximately 30 overlapping documents for inter-reviewer consistency. The audit focused on inspecting the deidentified outputs for apparent residual direct identifiers, including personal names, initials, facility names, addresses, telephone numbers, and patient IDs.

Because the analytic text explicitly reflects entities extracted by the NLP pipeline, this audit served as a sanity check of the generated output rather than a measure of general-purpose NLP performance on raw narratives. No apparent residual direct identifiers were detected in the audited sample; however, this does not exclude the possibility of residual direct identifiers outside the sample. For operational data release, we plan to implement a multilayer quality assurance process combining sample-based manual review with automated PHI screening using SQL-based pattern checks.

This study has several limitations. First, although the source text population consisted of all available linked progress note records in the eligible cancer cohort, conditional recoverability estimates (Layer D) are restricted to documents with at least one recoverable mention and would overstate recoverability; therefore, we also reported marginal extraction recoverability (document level: 30.7%; patient level: 36.3%) across the full downstream analyzable text cohort (Layer C). Accordingly, the reported recoverability metrics should not be interpreted as unbiased performance estimates for an entirely unselected corpus of all oncology progress notes.

Second, the internal reference evaluation annotation corpus was restricted to documents with character lengths between the first and third quartiles (Q1-Q3) within each document type to keep the annotation tractable within the governance-constrained budget. Entity-level metrics may not generalize to extreme-length documents.

Third, ICD-10 level-3 cancer categories with fewer than 50 patients were excluded from subgroup analyses (Table 2) to ensure stable estimates. Performance on rare cancer categories cannot be inferred from the present results.

Fourth, the NLP annotation corpus was small (100-document model-development set; 25-document hold-out). No comparison against rule-based systems nor off-the-shelf clinical NER benchmarks was performed; the IAA F₁ of 0.593 provides the primary human performance contextualization of the fine-tuned model F₁ of 0.557.

Fifth, the ATC code L Stage IV proxy includes neoadjuvant and adjuvant therapies for early- to intermediate-stage disease. The concordance analysis was exploratory. The early-stage procedure proxy had a very low PPV (0.031) due to an inadequate keyword list; it was retained as illustrative only.

Sixth, short median observation times (244.5‐297 d for major cancer subgroups) constrained the proxy-based concordance analysis. The time window sensitivity analyses (30/60/90 d) and incident/prevalent stratification characterized this effect.

Seventh, the masking rate (text opacity rate) quantified redaction volume and did not constitute a privacy guarantee. Calendar dates were retained by design; quasi-identifier risk for rare cancer patients was acknowledged. Risk management relied on governance-level access controls. Formal readability assessment of the deidentified analytic text was not performed.

Eighth, this study used data from 2 hospitals, which remain unnamed due to governance requirements, and generalizability across institutions and documentation styles has yet to be established. Generalizability remains an important consideration for Japanese clinical NLP, as prior work has highlighted the challenges of cross-hospital deployment even for well-defined extraction tasks [8,11]. Scaling to 50 or more sites will require continuous monitoring for distribution shift, site-stratified performance assessment, and governance frameworks for model updates and PHI quality assurance.

Ninth, because the production implementation includes proprietary components and was executed within a certified governance environment, executable code and real example data cannot be released publicly. To improve transparency as far as feasible, we provide a conceptual workflow description, masking policy categories, stepwise pseudocode (Conceptual Workflow for Deidentification and Entity Extraction section in Multimedia Appendix 2), and a synthetic side-by-side example illustrating how clinically relevant oncology information is retained in deidentified analytic text (Table 1).

Tenth, detailed corpus description statistics for the annotation partitions (total character count; average document length; document type distribution; and PHI entity class distribution across categories such as personal name, date, institution name, patient ID, address, and contact information) were not computed in this study. Such analysis requires additional processing of raw source text within ICI, which retains exclusive access to identifiable source notes and PHI-labeled entities under the governance framework separating upstream and downstream processing; this was outside the scope of this study and is noted as an opportunity for future methodological transparency.

Eleventh, the population selection algorithms defining the cancer cohort (ICD-10 C00–D48) and the downstream analyzable text cohort (Layer C 5-criteria inclusion rule) have not been independently validated against a patient-level gold standard or external electronic case report form review. Validation of these population selection criteria is acknowledged as a priority for future work when governance arrangements permit.

We developed a framework that delivers deidentified Japanese EHR narratives while preserving narrative structure for auditing alongside structured entity outputs. This approach addresses the privacy-reproducibility trade-off inherent in real-world data research. With high-yield linkage between narratives and structured records and cohort-specific evidence of feasible extraction and deidentification, the framework has the potential to enhance the efficiency and credibility of pharmacoepidemiologic analyses leveraging Japanese EHR free text and claims data.