Integrating Natural Language Processing in Child and Adolescent Mental Health Services Research Using Normalization Process Theory: Implementation Report

Introduction

Context

Anxiety problems are common among autistic children and young people and those with attention-deficit/hyperactivity disorder (ADHD), and their prevalence increases with the complexity of co-occurring autism and ADHD, representing a substantial public health concern [1].

Despite clear clinical needs, research evidence on effective treatments for anxiety problems in neurodivergent children and adolescents remains inconclusive. A network meta-analysis found that cognitive behavioral therapy (CBT) was associated with lower posttreatment anxiety severity in autistic children, although a similar effect was not evident when examining changes in scores from baseline [2]. In children with ADHD, evidence is limited and mixed regarding the impact of first-line stimulant medications on anxiety outcomes. Some small-scale studies suggest benefits, while others indicate potential exacerbation of anxiety [3]. Similarly, other interventions (eg, the nonstimulant medication atomoxetine, selective serotonin reuptake inhibitors, and psychosocial therapies, including CBT) have a promising but very limited evidence base, with findings of treatment benefits in these groups still mixed [3-5]. These gaps call for practice-based investigation of how anxiety is documented and managed in routine care for autistic young people and those with ADHD.

Problem Statement

Within the UK National Health Service (NHS) child and adolescent mental health services (CAMHS) setting, long wait times for neurodevelopmental assessments and mental health care indicate substantial pressure on services and leave service evaluation and implementation research deprioritized. Clinically relevant information is often recorded in free text rather than consistently represented in structured fields, making manual review impractical at scale. These factors are major barriers to translating research into practice [6]. This project therefore used natural language processing (NLP) to convert selected information in CAMHS electronic health records (EHRs) into structured variables for research on documented anxiety treatment and outcomes in neurodivergent children and young people.

Similar Interventions

Traditionally, evidence generation in CAMHS has relied on prospective research (randomized trials and observational studies) and retrospective service evaluations, which can be slow and resource-intensive. Controlled studies may not represent routine practice or heterogeneous clinical populations, while service evaluations typically measure end points (eg, service performance) rather than the practices and contextual factors that influence long-term success in real-world settings.

Approaches using routinely collected clinical data are now established as complements to prospective studies and manual service evaluations. Automated text mining and NLP have been used to extract clinical information from EHRs for research purposes in other domains of psychiatry [7-9]. However, little is known about how such tools are implemented, governed, and integrated within mental health services research settings. Normalization process theory (NPT) [10,11] conceptualizes implementation through coherence, cognitive participation, collective action, and reflexive monitoring and has been applied within an NHS context [12]. This implementation report therefore aims to document how an NLP-enabled information extraction (IE) workflow was specified, governed, embedded, and evaluated within NHS CAMHS research and to identify facilitators and barriers to its normalization using NPT.

Methods

Research Questions Informing the Implementation

The NLP workflow was developed to support an underlying clinical research program addressing the following questions:

• Do autistic children and young people, and/or children and young people with ADHD receive treatment for documented anxiety problems, and, if so, what treatments are recorded?
• What responses to these treatments are documented in routine clinical notes?
• What label-specific precision, recall, and F₁-scores does a locally deployed span-categorization model achieve when extracting the prespecified clinical constructs from EHR?
• How do documented outcomes vary across different treatment modalities (psychological, pharmacological, or combined)?

The results of this analysis will be reported separately. The original plan was to capture presenting problems strictly according to diagnostic criteria. This was revised because relevant information was frequently documented in formulation-led free text rather than as coded diagnoses or complete symptom lists.

Blueprint Summary

We implemented the NLP workflow through a 7-phase iterative road map (Figure 1) that combined technical development with integration into clinical research practices. First, the team defined the research questions. As a multidisciplinary team, clinicians and researchers identified questions about documented anxiety treatments and outcomes that could be addressed using available EHR data. As a result of this phase, the initial focus on formally coded anxiety diagnoses was broadened to include relevant manifestations documented in free text.

‎

Second, the team identified key constructs. The multidisciplinary team operationalized the definitions of presenting problems, types of anxiety interventions, and clinical response constructs such as symptom improvement, with side effects represented by a separate annotation category. Team members collaboratively and iteratively refined the list of key constructs and examples, thereby increasing stakeholder investment (“cognitive participation”).

Third, the team codeveloped the annotation schema. The constructs were translated into span labels with definitions, boundary rules, and disambiguation guidance. Pilot annotation identified additional concepts, including nonpharmacological support (eg, “buddy programs”). The final annotation schema, decision rules, and synthetic worked examples are provided in Multimedia Appendix 1; the schema was refined over 6 consensus rounds.

Fourth, the team trained the annotators. Four doctoral-level clinical researchers were trained to identify and label text spans according to the schema using an annotation platform. Training included written guidance, worked examples, and practical workshops, catering to nontechnical team members who faced a learning curve in using the annotation platform and understanding machine learning-oriented thinking (“cognitive participation challenge”).

Fifth, the team reviewed discrepancies. Preadjudication agreement was monitored using Krippendorff α. Annotators discussed disagreements in regular consensus meetings and workshops, after which definitions, boundary rules, and examples were clarified where required, providing opportunities for “reflexive monitoring.”

Sixth, the team annotated the EHR data. Over 4 calendar months, 4 clinical researchers reviewed and annotated 970 clinical notes through the Clinical Record Interactive Search (CRIS) platform. Each note was annotated by at least 2 researchers.

Finally, the team developed and iteratively trained the model. Consensus-adjudicated annotations were used as the reference standard for training a span categorizer machine learning model to identify and label clinical constructs. The final model was evaluated against a held-out test set of annotated notes using strict label-and-boundary matching, precision, recall, and F₁-scores.

The NPT framework informed implementation throughout the 7 phases. The team regularly discussed how our understanding (“coherence”) evolved, how engagement (“cognitive participation”) was fostered or impeded, what actions and operational work (“collective action”) were required, and how we appraised progress (“reflexive monitoring”). We also documented any adaptations made to the original plan as part of normalizing the new processes into our workflow.

Technical Architecture and Key Features of the NLP Tool

Processing Environment

The pipeline processed pseudonymized free-text clinical notes and returned structured records containing the predicted label, character-level start and end offsets, source note identifier, and associated metadata. All processing was performed within a trusted research environment.

Data Platform

Clinical notes were accessed through the CRIS system on the Akrivia Health platform [13]. The source data and NLP pipeline were hosted within the same secure Amazon Web Services (AWS; Amazon Inc) virtual desktop environment. Patient-level free text was not transferred outside the TRE.

Software

The pipeline was implemented in Python (version 3.10; Python Software Foundation; Multimedia Appendix 2 provides details on all software), using spaCy’s (Explosion AI) SpanCategorizer for supervised span identification and labeling [14], PyTorch (Meta Platforms Inc) [15] and Hugging Face Transformers for model training [16], and scikit-learn and custom Python code for evaluation. The evaluated medication, problem, psychological therapy, and outcome targets were all represented as span-categorization tasks; no document-level multilabel classifier was used. This in-house approach allowed the annotation schema and model configuration to be adapted to the study’s clinical constructs and secure computing environment. Previous work by the group informed the development approach [17-22].

Availability and Intended Use

The annotation schema is provided in Multimedia Appendix 1. Code and model configuration files are available to researchers with the required CRIS approvals within the secure environment. The pipeline was developed for secondary research and has not been validated for clinical decision-making. The intellectual property of the pipeline is owned by the University of Oxford and managed operationally by the Oxford Health NHS Foundation Trust (OHFT) CRIS team. Deployment in another organization or for clinical use would require external validation, assessment of generalizability, and local information governance and regulatory review.

Target Population and Setting

Implementation Target

The primary target of this NLP tool was the research process within an NHS CAMHS context. Specifically, researchers and staff aimed to improve the efficiency and scope of data analysis for a study on anxiety problems in neurodivergent youth. The intended beneficiaries of the knowledge gained are service users and clinicians working with children and young people with autism and/or ADHD and anxiety problems, by supporting subsequent governed observational analyses. The immediate users of the NLP tool are clinical researchers in our team who have access to CAMHS EHR data, as the tool was designed to enhance their capabilities in extracting and analyzing information.

Implementation Setting

This project was implemented at the OHFT, a public mental health care provider in England. OHFT’s services cover several counties (Oxfordshire, Buckinghamshire, Swindon, Wiltshire, Bath, and North-East Somerset), including community CAMHS clinics.

Impacts of the Implementation

The full annotation schema covered 4 broad information domains: presenting problems, treatment provision, outcome-related information, and working diagnoses [23]. The model evaluation reported here was restricted to 11 span labels covering medication attributes, problems, psychological therapy, and outcome-related information. Working diagnosis and 5 other schema labels were not included in the evaluated model. The evaluated domains were selected because they addressed the prespecified research questions and were not consistently available in structured EHR fields. Detailed analyses of documented treatment provision and outcome-related information will be reported separately.

Research Target

The intended study population comprised children and young people (aged 6-18 years) with evidence of autism and/or ADHD and evidence of anxiety difficulties [23]. To maximize sensitivity during initial record retrieval, the OHFT CRIS team first created a broad candidate cohort using a union criterion that contained evidence of autism or ADHD, evidence of anxiety, or both (Multimedia Appendix 2).

The present implementation study evaluates the development of the extraction workflow within the broad candidate cohort; analyses of clinical findings in the final both-criteria clinical cohort will be reported separately. The OHFT CRIS data team applied these criteria using SQL to identify the eligible cohort and retrieve associated pseudonymized clinical notes. The broad candidate cohort contained 454,127 notes from 5906 patients who received CAMHS care within the OHFT between 2011 and 2022. From this source cohort, 970 unique notes from 426 patients were selected for annotation using random sampling at the patient level. Ten notes from 7 patients were used for schema piloting and excluded from quantitative analyses. Of the remaining 960 notes, 263 notes from 117 patients were allocated to the interannotator reliability corpus and were independently reviewed by 4 annotators; the remaining 697 notes from 302 patients formed the model corpus.

Data Governance and Privacy

All data used (clinical notes and structured fields) were owned by the OHFT and accessed through the AWS environment used by Akrivia Health, a platform designed for secondary use of pseudonymized patient data. Researchers could only access EHRs via a secure virtual desktop infrastructure (VDI) on AWS, which prevented any patient identifiable information from being removed or exposed externally. Within this environment, all data processing (ie, manual annotation and automated NLP) was performed.

Data life cycle considerations were addressed as follows: clinical data were collected in routine care and then processed for research by Akrivia Health following an automatic pseudonymization step. The pseudonymized data were stored in the secure AWS workspace. No data were downloaded to personal computers. Researchers only accessed the pseudonymized dataset without modification to the source data. The OHFT CRIS team oversees all research projects and regularly audits data use on the Akrivia Health platform.

Interoperability With Clinical Systems

The Akrivia Health platform we used aggregates data from 2 EHR systems used in the Trust (ie, CareNotes [OneAdvanced] and RiO [The Access Group]). Akrivia Health itself handles the extraction and pseudonymization of data from those systems and provides data in a standardized format (both structured fields and text). Our text mining NLP IE pipeline was developed to process this CRIS-formatted data. In practical terms, this meant our data input scripts were tailored to the database schema and data tables of CRIS (eg, knowing how encounters, patients, and notes are linked via IDs).

NLP Model Development and Evaluation

Target Concepts

All reported targets were annotated as labeled text spans with character-level start and end offsets and were formulated as span-categorization tasks. Medication labels comprised drug name, dose, action, and frequency. Problem labels comprised cognitive, emotional, social, biological, and other problems, with other treated as 1 aggregated problem category. Additional labels identified psychological therapy and treatment outcomes. Table 1 summarizes the modeled labels and their frequencies. Additional categories retained in the annotation schema for descriptive annotation or future model development are listed in Table 1 and Multimedia Appendix 1. Technical details of model architecture and data handling are provided in Multimedia Appendix 2.

Model Architecture and Evaluation Protocol

All 11 targets were implemented using spaCy’s SpanCategorizer [14], with a ROBERTa (Meta AI)-based encoder [24], the standard pretrained weights of which were obtained through Hugging Face Transformers [16,24]. The SpanCategorizer and encoder were fine-tuned jointly on the consensus-adjudicated annotations in the model development partition; long notes were processed into paragraph-based chunks, and predicted offsets were mapped back to the source note. The 697-note model corpus was split at the patient level into 549 model development notes (from 226 patients) and 148 held-out test notes (from 76 patients), with no patient in both partitions; internal model selection used a note-level 440/109 split of the development notes and a fixed decision threshold of 0.5.

Interannotator reliability was estimated using position-based nominal Krippendorff α on the 263 notes reviewed by all 4 annotators. The held-out test set was used once to estimate strict span-level precision, recall, and F₁-score (with label and both character boundaries exactly matching a reference span), with 95% CIs from 5000 patient-level cluster bootstrap replicates. Full details are provided in Multimedia Appendices 2 and 3.

Regular NPT-informed structured team reflection and evaluation were facilitated based on: (1) an implementation journal documenting key decisions, changes, and rationales, coded prospectively according to the 4 NPT constructs; (2) minutes from regular team and annotator consensus meetings; and (3) quantitative process indicators, including annotation agreement, model performance, and timeline deviations. These records were reviewed deductively against the 4 NPT constructs of coherence, cognitive participation, collective action, and reflexive monitoring. Initial categorization was undertaken by HC and subsequently discussed with the wider research team. Differences in interpretation were discussed during team meetings. Key themes were recorded by HC after each team meeting in the implementation journal. Main findings were summarized in the following Implementation Process Outcomes subsection.

Ethical Considerations

This study used pseudonymized routinely collected EHR data from the OHFT CRIS system. The Oxford Health CRIS research database has approval from the South Central-Oxford C Research Ethics Committee (15/SC/0247). The present study was reviewed and approved by the Oxford Health CRIS Oversight Group. As the study involved secondary analysis of pseudonymized data within the approved CRIS research database framework, a separate project-specific NHS Research Ethics Committee application was not required. All analyses were conducted within the approved TRE. The examples in Multimedia Appendix 1 are synthetic composites; no verbatim clinical record text is reproduced.

Implementation (Results)

Outcomes of the Implementation

Annotated Corpus and Interannotator Agreement

The 960 notes from 419 patients comprised 697 notes from 302 patients used for model development and evaluation, and 263 notes from 117 patients assigned to interannotator reliability analysis. Category-specific reliability differed substantially across constructs. For nonmedication labels, Krippendorff α was 0.579 (95% CI 0.477-0.635) for emotional problems, 0.546 (95% CI 0.409-0.699) for psychological therapy, 0.516 (95% CI 0.280-0.688) for social problems, 0.418 (95% CI 0.271-0.467) for other problems, 0.411 (95% CI 0.318-0.526) for biological problems, 0.343 (95% CI −0.028 to 0.549) for outcomes, and 0.325 (95% CI 0.096-0.418) for cognitive problems. Medication label reliability was higher: 0.967 (95% CI 0.955-0.979) for drug name, 0.927 (95% CI 0.881-0.965) for frequency, 0.926 (95% CI 0.892-0.959) for dose, and 0.665 (95% CI 0.579-0.743) for medication action. All estimates are reported in Multimedia Appendix 3.

The model development and evaluation corpus contained 494,963 words; the median note length was 527 (IQR 261-925) words. Before model training, the 697-note model corpus was divided at the patient level into a 549-note model development partition and a 148-note held-out test partition. The complete model architecture and training configuration are specified in Multimedia Appendix 2.

Technical Evaluation Outcomes

Using strict label-and-boundary matching on the held-out test set, medication span F₁-score was 0.98 for drug names, 0.92 for doses, 0.91 for frequencies, and 0.66 for medication action (Table 2). For problem, psychological therapy, and outcome spans, label-specific F₁-score ranged from 0.60 to 0.86 (Table 3). Performance was highest for drug names and the aggregated other-problem label and lowest for cognitive and social problems, biological problems, medication action, and outcomes. These results demonstrate substantial variation across labels, which cannot be represented by a single overall F₁-score.

Strict span-categorization performance varied across the broader clinical constructs. Emotional problems and the aggregated other-problem label had the highest F₁-scores, whereas cognitive, social, and biological problems and treatment outcomes had lower performance. The wider CIs for several labels reflect greater uncertainty in the less frequent categories (Table 3).

In terms of efficiency, the model processed approximately 3 notes per second (Intel Xeon Platinum 8488C, 2.4 GHz, 32 GB RAM; Intel Corp), vs approximately 4 minutes per note for a human annotator, representing an approximately 700-fold increase in throughput, supporting practical viability for large-scale observational research. This operational comparison indicates substantially higher processing throughput than manual annotation under the reported environment.

Implementation Process Outcomes

Using NPT as a guiding framework, we summarized observations about how NLP was understood, enacted, appraised, and integrated within our team’s workflow (Table 4). No major adverse effects or unintended harms were reported.

Lessons Learned

Throughout this project, we identified the following strategies to address our main challenges (Table 5).

Discussion

This implementation report shows that an NLP-enabled workflow can be embedded within a secure NHS CAMHS research environment to convert selected information in free-text notes into structured research variables. Multidisciplinary schema development, consensus work, secure infrastructure, and ongoing appraisal supported implementation, whereas annotation burden, documentation variability, data access delays, and computing constraints required adaptation. Technical performance, treated as an implementation feasibility outcome, varied by construct: medication names, doses, and frequencies were annotated and extracted more consistently than medication actions, broader problem categories, and treatment outcomes. The resulting dataset is intended to support observational research; the pipeline has not yet been validated for clinical decision-making.

The evaluated pipeline represents a preparatory IE component rather than a complete treatment response analysis. Because medication, attribute, therapy, and outcome spans were extracted independently, answering the downstream clinical questions will require additional note- or patient-level aggregation, resolution of treatment attribute and treatment outcome relationships, and validation of the resulting derived variables.

The road map and NPT-informed documentation provided transferable lessons for integrating advanced analytics within research practice. Mapping activities and observations to coherence, cognitive participation, collective action, and reflexive monitoring showed where implementation progressed, encountered barriers, and required adaptation. Repeated pivots also illustrated the value of an iterative, learning-oriented approach, rather than being considered as implementation failures.

Several limitations should be considered. First, this was a single implementation within 1 NHS Trust, so the transferability of findings may be limited by differences in populations, documentation practices, infrastructure, and governance. We followed the iCHECK-DH (Guidelines and Checklist for the Reporting on Digital Health Implementations) reporting guidance [25] to support transparency (Multimedia Appendix 4). Second, the NPT-informed evaluation relied partly on team-generated records and interpretations, and stakeholder acceptability and feasibility were not measured using validated instruments. Third, documentation and annotation ambiguity varied by construct. Medication names, doses, and frequencies generally had clearer lexical boundaries than problem categories and treatment outcomes. Category-specific Krippendorff α ranged from 0.325 to 0.579 for nonmedication constructs and from 0.665 to 0.967 for medication attributes; for outcomes, Krippendorff α was 0.343 (95% CI −0.028 to 0.549), indicating substantial uncertainty and not excluding chance-level agreement. Lower reliability may reflect both ambiguity in the clinical constructs and their documentation, as well as inconsistency in applying the operational definitions. Model performance measures agreement with the consensus-adjudicated reference annotations and should therefore be interpreted in light of these sources of ambiguity; it does not directly quantify construct ambiguity or establish clinical validity. Fourth, the 549-note model development partition was split at the note level, so notes from the same patient could occur in both the training and internal development subsets, although the held-out test set was patient disjoint. Fifth, the model was trained once using a single random seed. The patient-level bootstrap CIs reflect patient resampling in the test set, not training variability. Finally, the model evaluation remained local and research-only, without external or clinical validation for nonresearch purposes (eg, as a clinical decision tool). The model extracted spans independently and did not link medication attributes or outcomes to treatments or classify outcome polarity or temporality. No dictionary-based, rule-based, or alternative machine learning baseline was evaluated; the results therefore establish local feasibility and label-specific performance, but they do not imply superiority over simpler extraction methods or alternative model architectures.

In conclusion, an NLP-enabled workflow can structure selected mental health information from clinical records at scale, but implementation requires multidisciplinary co-design, iterative definitions, realistic resourcing, secure infrastructure, and ongoing appraisal. Refinement of the definitions and extraction methods for less reliable labels, and evaluation in other settings are required before broader use.