To create the NLP model to assess the adoption of PPC, we used the notes of a cohort of social workers who were trained to be PPC clinicians over the course of 9 months. This cohort of social workers was hired through the VA Social Work in Patient Aligned Care Teams (PACT) Staffing Program. This program was developed in May 2016 and is the result of a partnership between the Veterans Health Administration (VHA) National Social Work Program Office and the Office of Rural Health. The goal of the program is to embed social workers in rural and highly rural areas to increase Veteran access to high-quality social work services. The innovative program supports the Office of Rural Health’s mission of breaking down barriers separating rural Veterans from high-quality care. Social workers are often one of the first clinicians a Veteran has contact with in the VHA. Social workers in this program provide interventions at numerous rural health VHA medical centers and outpatient clinics across the nation. Two cohorts of social workers were trained: 56 social workers from 5 VA health care systems in 2018 and 88 social workers from 12 VA health care systems in 2019. Social workers were trained to assess, evaluate, and select patients who were good candidates for PPC. Patients had to be older than 60 years of age and have at least two primary care encounters in a prior year. Social workers were trained to facilitate PPC conversations using patient and facilitator workbooks. Social workers were instructed to document all PPC-related interactions in the patient free-text note including all components of PPC. A PPC rubric was provided to social workers to document components of PPC: patient values, outcome goals, and health care preferences (see Multimedia Appendix 1). After reviewing the PPC template and related notes, we recognized that all notes must have 2 phrases: “goal” and “value.” Notes without these 2 expressions are not considered as PPC.

Notes of eligible patients were retrieved from the VHA Corporate Data Warehouse [13]. First, notes of all patients who visited with a PPC-trained social worker were retrieved from the Corporate Data Warehouse and included in a database. Notes were obtained from the first facilitated PPC visit through the end of the 2019 calendar year. The Baylor College of Medicine Institutional Review Board and the Veterans Affairs research service (IRB# H-41886) approved this study. Access to patients' data was granted through the Veterans Informatics and Computing Infrastructure.

To develop the NLP algorithm, we extracted 778 notes from 658 patients who were seen by the social workers. We developed a reference standard for identifying PPC using a formal chart review. We provided complete copies of the notes from the 2018 and 2019 cohorts to 2 independent reviewers (EO and JF). The reviewers ensured the existence of language for values, outcome goals, and care preferences for each free-text note. Charts were labeled as 0 or 1; 0 refers to absence of documented patient priorities for the patient’s note, and 1 refers to documentation of priorities in free-text notes of patients. Any disagreement between reviewers was resolved by a third reviewer, a geriatrician involved in the development of PPC (AN). The agreement among reviewers is reported using the kappa statistic, and the significance level was set at P<.05; statistical analysis was conducted using SPSS software.

The NLP algorithm was developed using 3 steps: preprocessing, processing, and postprocessing (Figure 1). In the preprocessing step, we cleaned the text, performed features extraction, and reduced the number of features to reduce the size of the feature space (dimensionality reduction). In the processing step, we developed a classification model to identify the yes-PPC from no-PPC notes. In postprocessing, we reported the performance models developed in the processing step and assessed the generalizability of the model.

As a second objective, we evaluated the adoption of PPC among trained social workers. We counted the number of yes-PPC notes in each month, and we created a time series of notes identified by the PPC algorithm. The x axis shows the month, and the y axis shows the number of documented notes with PPC language in the EHR. This evaluation helped to understand the implementation of the PPC approach among social workers who received formal training and the persistence of PPC adoption over time.

We collected 778 notes that had “goal” and “value” within their text. These notes were collected from 658 patients from 17 VA facilities by 82 social workers. Mean patient age was 70.4 years (SD 15.2 years), 91.6% (603/658) were male, 23.4% (154/658) had all-cause-of-mortality recorded, 91.8% (604/658) were of “Not Hispanic or Latino” ethnicity, 71.9% (473/658) were White, 6.7% (44/658) were “Black or African American,” and 47.4% (312/658) were married.

The 2 reviewers disagreed on 54 notes and agreed on 724 (724/778, 93.1%) notes. The level of agreement after removing the chance effect was reduced to 82% (kappa=0.820, standard error 0.023, P<.001).

The performance of the NLP model is shown in Table 1. The NLP model performance was calculated over notes (N=778) that had “goal” and “value” in their text. In the 10-fold cross-validation, the model validation reached an average of 0.92 (95% CI 0.90-0.94) for accuracy, 0.84 (95% CI 0.79-0.89) for sensitivity, 0.95 (95% CI 0.92-0.97) for specificity, 0.84 (95% CI 0.77-0.91) for the positive predictive value, and 0.95 (95% CI 0.93-0.96) for the negative predictive value.

In contrast, an approach using simple search terms (“goal” and “value”) applied to large text performed less well. The total number of notes that the social workers produced was 106,505. Table 2 describes the precision of this model. The precision of the NLP model was >0.99, while the precision of the simple search terms was 0.7571. In the other words, the simple search terms method falsely recognized 1 out of 4 notes as PPC.

Figure 2 describes the adoption of PPC within the notes of trained social workers over time. The 2 peaks represent the times immediately following training of social workers to use PPC. Figure 2 also demonstrates that several months following the cessation of training, the persistence of PPC adoption declined. The NLP algorithm was useful in tracking PPC adoption over time.

Our results demonstrate that an NLP algorithm was highly accurate in identifying patients’ free-text notes with PPC language. Such a design has shown promise in previous work identifying risk factors for heart disease [18]. The NLP model had a high performance of 0.84 in validation, and its precision was higher than an approach using only simple search terms. The NLP algorithm was also able to capture peaks in PPC adoption following social worker training and the eventual decline in PPC adoption over time. Such a tool may be used to monitor the overall implementation of interventions like PPC among trained social workers and to adjust the training process.

While this NLP model performed well, model performance (precision and recall) could improve with the use of more complex strategies such as utilizing bigram or n-gram BoW, other sophisticated feature extractions (eg, graph mining [19]), and nonlinear models (eg, deep learning [20]). To reduce the computational cost of developing and implementing a model to test the use of PPC by trained facilitators, we propose the NLP model as a method that reduces the time and cost with other methods of assessment. Further development of this model will offer assessment with more precision. Despite these limitations, the NLP model performed better than the simple search term method, which had a 1-in-4 notes rate of misclassification.

This study opens 3 new avenues for further investigation. By identifying PPC notes, we can longitudinally assess care for patients throughout VA facilities and understand changes in health care delivered after PPC encounters [6,21]. We can identify patterns and consistency in how social workers and other clinicians document patient priorities over the course of multiple interactions with patients [3]. By further expanding the NLP algorithm, we can also examine common priorities among our patient population and increase clinician, patient, and caregiver awareness of priorities recorded in the EHR. For successful implementation of PPC, we must assess other factors that play roles in the successful implementation of PPC, such as resource availability, adoption of PPC on a provider and facility level, and readiness of clinicians for implementation [4]. We can also use sociodemographic data to pinpoint the ways individual characteristics of patients might influence the implementation of PPC by clinicians. We can also provide individualized feedback to each clinician to enhance their patient engagement process. Using this algorithm, we can measure long-term outcomes of interest, such as mortality and independent living.

This study achieved its overall objective of developing a simple NLP model to identify free-text notes that show evidence of documented patient priorities as part of the PPC process. Our NLP model complements PPC by measuring the extent to which social workers trained in PPC identify and document patient priorities. The NLP model can enable trainers to provide feedback on the implementation of PPC in the EHR. Additionally, this method can analyze the free text of any note for documentation of patient priorities generally in an effort to link those documented priorities with many patient outcomes without requiring labor-intensive methods for individual chart review.