The Impact of Collaborative Documentation on Person-Centered Care: Textual Analysis of Clinical Notes

Introduction

Collaborative documentation (CD) is a specified behavioral health practice where clinicians complete visit notes jointly with consumers during the session [1]. Through deliberate clinical strategies, such as sharing the computer screen, reading visit notes aloud, and actively seeking consumer’s input into the content of the session note [2], CD is a person-centered strategy that aims to engage and empower individuals and facilitate a mutual agreement on treatment progress, service goals, and session activities. Both as a means to promote person-centered care (PCC) and make health information more accessible and transparent to consumers, the practice of CD is being widely disseminated through formal and informal training for community mental health providers [3].

PCC, also referred to as patient-centered care, is a paradigm shift in health care that is defined by the Institute of Medicine as care that is responsive to individual client preferences, needs, and values [4]. A key part of behavioral health reform, PCC moves away from disease-centered treatment to a more holistic approach that engages individuals as active, empowered partners in their care. Person-centered care planning (PCCP) is a recovery-oriented practice that integrates principles of PCC into the service planning process. PCCP orients service planning and documentation to the unique personal life goals of the consumer [5], and provides a framework for operationalizing PCC in practice through six core principles, which are as follows: (1) PCC is based on the person’s own unique life goals and aspirations; (2) PCC is oriented toward promoting recovery rather than only minimizing illness and symptoms; (3) PCC articulates the person‘s own role and the role of both paid practitioners and natural supports in assisting the person to achieve his or her own goals; (4) PCC focuses and builds on the person’s capacities, strengths, and interests; (5) PCC emphasizes the use of natural community settings rather than segregated program settings; and (6) PCC anticipates and allow for uncertainty, setbacks, and disagreements as inevitable steps on the path to greater self-determination [6].

While CD has emerged as a recognized person-centered practice strategy by fully engaging the consumer in decisions about their care, there remains a very limited evidence base demonstrating its clinical effectiveness, including its impact on PCC. Existing research demonstrates that CD is aligned with consumer preferences; a recent scoping review found that the ability to read their visit notes improved consumers’ experience in mental health care, including their ability to remember their plans of care, understand their treatment, and trust decisions made with providers [7]. In community mental health, the use of CD specifically has been found to strengthen the therapeutic alliance [8], and improve service engagement, both in terms of visit attendance and medication adherence [1].

Despite this preliminary support for CD, more work is needed to examine its impact on quality of care. Adding urgency to this knowledge gap, regulatory changes to the 21st Century Cures Act now mandate organizations to make electronic health and mental health information, including many types of visit notes, accessible online to service users [9]. As a consequence, best practices for using visit notes to support the provision of high-quality PCC are needed.

Clinical informatics, which provides highly efficient ways to mine data within the electronic health record (EHR), is a promising methodological approach to examining the impact of CD on clinical quality. Although behavioral health has lagged behind medical settings in the adoption of EHRs, now the majority of behavioral health settings document visit notes via the EHR [10]. The shift from paper records to EHRs provides an unprecedented opportunity for clinical informatics to inform quality improvement in behavioral health care. Visit notes include nuanced information about care processes, session content, provider perspective, and the consumer experience that are not captured in other more structured fields of the EHR, offering valuable insight into clinical quality that has not yet been systematically targeted in mental health services research [11]. Researchers have applied manual content analysis to visit notes to evaluate dimensions of PCC [12,13], but these efforts are inevitably limited in scope and dependent on the interpretive lens of the researcher [14].

One clinical informatics strategy that can parse large volumes of unstructured narrative information into quantitative data is natural language processing (NLP) [15]. While some text mining approaches use words as the unit of analysis, NLP is able to capture the complexity of unstructured narrative using underlying metadata, which examines how words relate to each other in a sentence and the semantic context of a sentence [16]. The method involves syntactic processing, information extraction, and capturing meaning and relationships across concepts. NLP has been predominantly used for detecting pathology and predicting behavior [17] including measuring the following: alcohol misuse in trauma patients [18], suicidal behavior [19], adverse childhood experiences among VA patients [20], smoking status [21], and sentiment at discharge [22]. Recent studies have used this method to measure quality and safety in nursing care [23], identify integrated care elements within primary care [24], and detect changes in clinical documentation after opening notes to service users [25], however, there have been fewer studies that have used NLP to analyze indicators of mental health care quality, including PCC. By providing a framework to systematically categorize and compare the contents of clinical notes, NLP is well poised to provide novel insight into how CD affects clinical quality and PCC.

The dictionary-based approach in NLP is a method that uses a predefined lexicon to identify and extract certain types of words or phrases within a given text. This approach is often used in tasks such as part-of-speech tagging, named entity recognition, and sentiment analysis. In the dictionary-based approach, the dictionary consists of a list of words or phrases along with their associated tags or labels. For example, in part-of-speech tagging, the dictionary might contain a list of common nouns, verbs, and adjectives, each with a corresponding tag that indicates the word’s part of speech. One of the benefits of the dictionary-based approach is its simplicity and ease of implementation, as it targets only a predefined lexicon, or set of words, and does not require the use of complex algorithms or machine learning models. One of the most dictionary-based approaches is LIWC (Linguistic Inquiry and Word Count), and LIWC analysis has been found to be particularly useful for identifying and analyzing the emotional state of individuals in mental health–related text data [26]. Despite its potential, such as its application in oncology settings to examine the changes in clinical notes after patient access [27], the dictionary-based approach has not yet been applied to clinical notes in behavioral health care settings for its change after CD.

This study examines the effect of CD on the person-centeredness of documentation within a community mental health setting using a quasi-experimental pretest-posttest design. The study adapted a well-established dictionary to conduct sentiment analysis of provider clinic visit notes before and after providers were trained in CD.

Methods

Ethical Considerations

This study (IRB-FY2022-5838) was granted an exemption by the New York University Institutional Review Board.

Data Source

The study setting was a community mental health center, which provided a range of services to people with severe mental illnesses including outpatient therapy, assertive community treatment, community support programs, and psychiatric rehabilitation. The data set, or corpus, is visit notes completed by providers trained in CD. Providers were trained in CD by MTM Services [28], a leader in CD training. Their training consisted of tailored in-person workshops, technical assistance, and a “train the trainer” series designed for practice sustainability. The clinic training consisted of 8 hours of web-based training and customized consultation support for implementation.

The visit notes were clinical narrative documents completed when the provider had an in-person contact with a service user. Visit notes document the following: (1) sessions focused on developing or revising the service plan, a narrative clinical document completed every 6 months with the service user detailing goals, strengths and barriers, short-term objectives, supports, professional/ billable services, and natural support and self-directed actions, and (2) a visit focused on progress made towards completing the steps in the service plan. Inclusion criteria for providers were that they were: full-time employees of the clinic; provided services to adults with severe mental illnesses; trained in CD; and had been employed in their position a year prior and the year after being trained.

This study sampled all visit notes completed by participating providers 6 months prior to CD training and all visit notes completed during the sixth month following training (with a month lag for implementation time) between July 1, 2015, and March 10, 2020. Based on anticipated documentation rates of a clinic note being generated by each visit, we recruited 10 providers, generating a total of 1981 visit notes. On average, 198.1 notes were included per provider with a standard deviation of 37.8. Sampled notes were deidentified but linked to providers through unique identifiers. In addition to the session narrative, each note was comprised of the following sections: (1) Provider ID, (2) Time and Date of Service, (3) Therapy Modality, and (4) Length of Stay.

Analytic Strategy

We used LIWC-22 to compute the scores for PCC-related sentiment and linguistic categories [29] in each sampled note. The LIWC method is a text analysis tool that uses linguistic algorithms to identify and categorize words in a text according to their psychological properties. The method is based on the LIWC dictionary, which contains over 5000 words and word stems organized into linguistic and psychological categories. In this study, 3 domain experts used a consensus approach to select LIWC categories that mapped onto the 6 principles of PCC [6] with several of the categories mapping onto more than one domain. This iterative process involved 8 rounds of independent coding by 3 raters using Excel, followed by discussions to resolve discrepancies and achieve consensus on the final coding of all data points. According to the Cohen κ measure, raters had an average interrater agreement of 88% across the LIWC categories [30]. Out of a possible 107 LIWC categories, we selected 12 categories. The selected LIWC domains and their associated PCC principles are summarized in Tables 1 and 2 [6,30].

Analysis

Using the LIWC categories described above, we compared notes pre and post CD training to examine differences in PCC. The complete visit note was used as the unit of analysis. For data preprocessing, we cleaned the data and converted the information into a structured format that made it amenable to identifying patterns in the data. The LIWC-22 dictionary is case-insensitive and allows for matching 2-word phrases. The LIWC-22 software removes extra whitespace characters by default. While irrelevant words are not explicitly defined in the dictionary, we removed section headings (eg, Location) from the clinical notes before processing the text to eliminate some irrelevant words. Negated phrases (eg, “not happy”) are not treated differently from nonnegated phrases in the standard scoring. To address this limitation, we included an additional analysis in Multimedia Appendix 1 that controls for the negations score.

To validate the team’s selection of LIWC categories, we first used the Contextualizer function of LIWC-22 to generate sentence fragments containing words related to each LIWC domain included in the analysis. We then analyzed changes in the clinical note before and after CD training, using the complete visit note as the unit of analysis. To calculate changes in the content characteristics of clinical notes before and after CD training, we calculated frequency scores of each LIWC category for every clinical note since this study focuses on examining the presence of words from the LIWC dictionary.

Instead of using LIWC scores based on percentages, we used frequency scores. This decision was made based on previous research indicating that CD enhances the length of clinical notes in terms of word and character count [31]. Additionally, we used frequencies to assess the presence of PCC-related language in clinical notes. Our primary interest was to capture whether clinicians used PCC-related words in their documentation, even if these words did not constitute a large proportion of the total text. By focusing on word frequencies, we aimed to mitigate the potential impact of note length inflation due to CD practices, such as copy-and-pasting or using templates [32,33]. The LIWC frequency scores are calculated by the following steps. First, we calculated LIWC scores, which are determined by the percentage of words in a text that belong to specific linguistic categories. Then, to find the frequency of each category within a clinical note, the respective LIWC percentage is multiplied by the total word count of the note. For example, an LIWC value of 1.02 for achievement indicates that the note contains 1.02 words related to achievement per the LIWC dictionary.

We used a fixed effects model that included the provider as a categorical variable. The changes in the notes before and after CD training were calculated while accounting for nesting within the therapist using individual fixed-effects models. This approach allowed us to examine whether the changes in PCC language use before and after the CD training varied across the 10 providers in our sample. To further investigate these differences, we calculated the intraclass correlation coefficient (ICC) and conducted paired sample t tests for each provider (Multimedia Appendix 2). The LIWC version 22 was used for dictionary-based sentiment analysis and STATA (version 17.0; Statacorp) was used for statistical analyses.

Results

Contextualizing LIWC Categories

The sentences generated by the Contextualizer function illustrated how the LIWC categories mapped onto the PCCP principles (see Table 2). The drives subcategories (achievement, affiliation, and power) reflected a strength-based approach by describing the positive changes made by the client; greater self-determination by feeling more in control; and interests by capturing how a client feels connected to community, people, and pets. The lifestyle subcategories (home, leisure, money, religion, and work) captured the unique details of the person’s life that are needed to individualize treatment, including their beliefs, values, and preferences which inform their personal life goals. Examples included going on a cruise as part of self-care and seeking employment. Lifestyle categories also illustrated people’s interests such as playing sports and describing their life in the community such as attending church. Physical categories (physical and wellness) demonstrated a more holistic approach to the client by paying attention to how physical health affects mental health and also to a focus on recovery by including activities that promote wellness such as yoga and health coping skills. The social referents category (family and friend) demonstrated the role family and friends play as natural supports such as going on vacation with your sister or having fun with your friends.

Changes in LIWC Categories

Overall, there was a significant positive change in 4 of the selected LIWC categories indicating person-centeredness after the providers had been trained in CD. As shown in Table 3, the fixed effects regression analysis found the following among the 12 selected characteristics: an increased use in 4 categories, decreased use in 4 categories, and no change in use in 4 categories, while controlling for length of sessions at the therapist level. Within the drives category, we observed a significant increase in words associated with achievement (β=.774, P<.001, ICC=0.146) and power (β=.831, P<.001, ICC=0.072), with the ICC values indicating that 14.6% and 7.2% of the variance in these word categories, respectively, could be attributed to differences between therapists. In the lifestyle category, there was an increase in the use of words related to home (β=.047, P=.35, ICC=0.060) and work (β=.047, P=.12, ICC=0.285), but these changes are not statistically significant. The ICC values suggest that 6.0 and 28.5% of the variance in these word categories, respectively, could be attributed to differences between therapists. On the other hand, leisure and religion-associated words showed a significant decrease (β=−.166, P=.002, ICC=0.033; β=−.105, P<.001, ICC=0.031, respectively), while words associated with money displayed substantial increases (β=.204, P<.001, ICC=0.035). The ICC values for these categories indicate that 3.3%, 3.1%, and 3.5% of the variance, respectively, could be attributed to differences between therapists. In the health category, there was a notable increase in the use of physical health–related words (β=.427, P=.03, ICC=0.159), with 15.9% of the variance attributable to differences between therapists. In contrast, wellness-related words decreased significantly (β=−.427, P<.001, ICC=0.211), with 21.1% of the variance attributable to differences between therapists. In the social referents category, the use of family-related words did not show any significant change (β=−.016, P=.89, ICC=0.085), with 8.5% of the variance attributable to differences between therapists. However, the frequency of friend-related words decreased significantly (β=−.084, P=.005, ICC=0.028), with only 2.8% of the variance attributable to differences between therapists.

Discussion

Principal Findings

The contextualizing analysis provided insight into the documentation content reflecting the selected LIWC categories and demonstrated how person-centered principles can be integrated into clinical documentation. Using LIWC categories illustrated how providers described their clients in ways that gave a sense of their lives beyond their mental health. These details included what they care about and how that related to their personal goals (getting a job, financial situation, going on a cruise, or attending church), their life beyond the clinic in the community (their home life, family and friends, and their community). The sentences also showed when clinicians used a strengths-based approach, the nature and content of their clinic notes changed in ways that moved beyond symptoms [34].

The quantitative results indicating whether there was an increase in person-centeredness of clinical documentation as indicated by relevant LIWC categories were mixed, with a significant increase in half the subcategories. The most pronounced positive increase was within the drives category, with words associated with power and achievement increasing. In terms of PCC, this indicates providers made more reference to self-determination, including how the client has made progress, and their strengths. Lifestyle categories, which include words related to hobbies and other social activities were more mixed, showing that providers were not consistent in increasing their focus on personal life goals or taking a holistic view of the client. In health, there was a significant increase in references to physical health but a decrease in references to wellness. This may reflect the increasing efforts to integrate health into their clinical interventions [35], but does not indicate a more recovery-oriented focus. Finally, in terms of social referents, there was no change in family references, which may be due to the fact that family inclusion is a common best practice of PCC [36,37], and a decrease in references to friends, often considered a source of natural supports within PCC.

Existing work has suggested that CD can improve important indicators of PCC, including service engagement and the quality of the working alliance [1,8], but research has yet not illuminated how this practice impacts care processes, including how person-centered principles are integrated into clinical interventions. Through analysis of session notes, this study found an increase in strengths-based approaches to clinical documentation following CD training, which may also reflect a shift towards interventions that emphasize self-determination. In addition to expanding the limited evidence base around the impact of CD on clinical quality, this study uniquely describes the mechanisms through which CD supports alliance building and engagement in care.

To meet the much-documented challenges of measuring PCC [38,39], a core component of health care reform, this study sought to harness the richness of clinical narrative data using a machine learning algorithm. While some studies have used NLP to study psychotherapy sessions [25], this is one of the first to mine narrative psychosocial documentation in behavioral health settings. Overall, this methodological approach has considerable potential to sample large quantities of documentation to measure different aspects of care quality, including PCC. NLP can be used both by researchers to examine how quality of care predicts clinical outcomes and by clinics to promote and document quality improvement.

Despite this potential, there are considerable challenges in calibrating algorithms so that they can accurately capture more nuanced aspects of care, such as person-centeredness. This study chose to use a well-validated dictionary designed to capture psychological concepts within narrative data. Although the study team was able to map existing LIWC categories onto established principles of PCC, the algorithm was not explicitly designed to measure these constructs, which may have contributed to the lack of positive findings within certain subcategories. Furthermore, the LIWC dictionary’s focus on single words limits the algorithm’s ability to capture more nuanced meanings that occur when words are evaluated within the larger context of surrounding phrases or sentences. This suggests the need to develop an algorithm focused specifically on PCC.

This study showed the potential for using NLP techniques to measure the quality of care within behavioral health settings. As more care standards demand that clinics demonstrate PCC [40] within mental health, there is a pressing need for feasible methods to capture this quality dimension. Being able to use at the aggregate level note data that reflects more nuanced and individualized aspects of care would help clinics document and report PCC.

In addition, our study highlights the value of clinical notes for research in behavioral health settings. Clinical notes have been harnessed for research purposes but have mainly been confined to hospital settings due to the scarcity of publicly available data [41]. The nature of psychosocial documentation differs from other clinical notes, which require different analytics and models and there is a need for publicly available data sets for analyzing psychotherapy notes in the United States. Furthermore, mental health notes often contain identifiable and sensitive information different from other clinical notes for physical illnesses, so an in-depth discussion on ethics, privacy, and deidentification and the development of techniques such as word embedding models to improve the privacy of clinical notes [42,43] for mental health notes is required.

Limitations

The study was limited to the scope of the LIWC categories rather than an algorithm developed specifically to capture PCC and therefore, was not able to measure the concept in its entirety. Furthermore, the analysis limited itself to categories with a positive valence rather than also measuring the inverse of PCC. Although well validated, the LIWC can still fail to capture the meaning of words and mis-categorize them but as it is used more, the algorithm will continue to be trained and improved. Overall, while documentation is an important indicator of PCC, it does not directly capture the interpersonal interactions between the provider and the clinician which shape a client’s experience of PCC.

While our fixed effects model controlled for Provider ID and Date of Service, we were unable to account for potential differences in therapy modality or length of stay. In our study setting, the therapy modality primarily consisted of individual psychotherapy sessions excluding 2 notes (1 group therapy and 1 family therapy) resulting in insufficient variation to control for this variable. Additionally, our current data set did not include sufficient information on length of stay to include this variable in the model. Future research could benefit from examining the influence of therapy modality and longitudinal factors on PCC language use in clinical documentation.

It is important to acknowledge that while our study demonstrates an increase in person-centered content within clinic visit notes following the implementation of the CD, we did not directly assess whether the increased PCC in the clinic visit notes was associated with improved PCC practices. Future research should investigate the relationship between the presence of PCC in clinical notes and its impact on PCC practices and outcomes.

Conclusion

This study is an important first step in using NLP to measure the quality of care through narrative clinical notes in behavioral health settings. We were able to identify key PCC principles within the notes using a dictionary-based approach and examine whether CD changes the way providers document with respect to PCC. This demonstrates the potential for NLP to be used by both researchers and clinics as a quality improvement tool and the importance of further developing algorithms that can capture the nuances of PCC.