Digital mental health tools, including therapeutic chatbots, are increasingly deployed to address rising psychological distress among young adults and the limited availability of traditional counseling services [1-3]. In South Korea, the proportion of young adults aged 20‐30 at risk for depression reached approximately 30% in 2021, representing a sixfold increase compared to 2018 [4]; similarly, anxiety and depression symptoms among college students in the United States increased by 75% between 2019 and 2021 [5]. Despite this growing need, treatment-seeking rates remain critically low, with fewer than 36% of distressed students using available counseling services. Evidence suggests that conversational agents can provide accessible and scalable support, demonstrating small but meaningful effects in university populations [2,3]. As deployment expands, health care organizations face growing pressure to maintain clinical safety, therapeutic appropriateness, and systematic quality assurance processes comparable to other clinical decision support tools.

However, a persistent evaluation–enhancement gap limits responsible iterative development. Although therapeutic chatbots are routinely evaluated through user studies, expert reviews, or large language model (LLM)-based assessments, these evaluations rarely translate into structured and governable improvement processes. Existing workflows often rely on ad hoc prompt adjustments or informal fine-tuning, leaving organizations without systematic methods to act on diagnostic insights, verify improvements, or document decision pathways for organizational oversight. This gap is particularly concerning in mental health contexts, where deficits in personalization, probing skills, or crisis handling can directly affect user safety. Although recent advances in multidimensional evaluation, think-aloud protocols, and LLM-assisted scoring have improved diagnostic precision, these frameworks remain primarily diagnostic and offer limited operational guidance on how identified weaknesses should be systematically enhanced and validated.

Our previous study [6] established a comprehensive baseline evaluation of Dr.CareSam, a GPT 4.0–based bilingual mental health chatbot. That study identified strong relational competencies, including positivity and support, empathy, and active listening, while revealing systematic deficits in professionalism, content complexity, and personalization. However, the evaluation concluded without providing operational pathways for systematic improvement, exemplifying the evaluation–enhancement gap in therapeutic artificial intelligence (AI) development.

We conducted a within-subject, participant-blinded A/B pilot validation study of the EvaluationPlus framework. Each participant interacted with both the baseline system (version A) and the enhanced system (version B) of Dr. CareSam, a bilingual GPT 4.0–based therapeutic chatbot (Figure 1). Sessions comprised 5‐6 conversational turns per scenario; participants accessed both versions independently via an online platform, and presentation order was self-determined rather than experimentally controlled. The 3-stage enhancement and validation process is illustrated in Figure 2.

The study was approved by the Sungkyunkwan University Institutional Review Board (IRB number 2023-02-043-007). All participants provided written informed consent and were informed that Dr. CareSam is a research prototype. Transcripts were fully deidentified per Korea’s Personal Information Protection Act (PIPA). A structured debriefing protocol was implemented after each session; participants engaging with the suicidal ideation scenario received written materials and national crisis helpline resources. No participants required clinical referral. Participants were compensated ₩20,000 (~USD 15) for approximately 60 minutes.

Korean graduate students were recruited at the Department of Applied Artificial Intelligence, Sungkyunkwan University. Inclusion criteria were Korean fluency and self-reported comfort with mental health content; participants in active crisis care were excluded. Sixteen participants enrolled; one was excluded under a prespecified dual criterion: statistical extremity (z=−2.60, |z| > 2.5 SD) and qualitative–quantitative inconsistency indicating task misunderstanding (full documentation in Multimedia Appendix 1). The final cohort was N=15 (Table 1). Data collection was conducted in August 2025.

Four standardized scenarios spanned a clinical severity spectrum: presentation anxiety (mild), academic dropout ideation (moderate), panic symptoms (moderate–high), and suicidal ideation (crisis) [18] (Figure 3). User validation used three scenarios (presentation anxiety, academic dropout, and suicidal ideation); participants completed a minimum of 2 per version. The panic scenario was reserved for expert evaluation only. Across both versions, 13 of 15 participants (86.7%) engaged with Scenario 4 (suicidal ideation), 9 (60%) with Scenario 1 (presentation anxiety), and 3 (20%) with Scenario 2 (academic dropout). Bilingual scenario scripts are provided in Multimedia Appendix 2.

EvaluationPlus integrates 3 sequential stages (Figure 2).

A multistakeholder framework integrated 3 evaluator perspectives (Tables 2 and 3). Full evaluation instruments are provided in Multimedia Appendix 2.

All analyses followed a prespecified within-subject plan emphasizing effect size estimation over null-hypothesis testing, consistent with pilot validation practice. As a pilot feasibility study, the sample size was determined to permit intensive multiscenario evaluation (each participant generated 14 independent dimension-level ratings [7 dimensions×2 versions]; where participants engaged with multiple scenarios, dimension ratings were averaged across scenarios prior to analysis to yield a single composite score per dimension per version) rather than to achieve formal statistical power.

Descriptive statistics (mean±SD) were computed for both versions across all seven dimensions. The primary effect size was paired Cohen dz = (Mᴮ − Mᴬ) / SDΔ. Enhancement targeting validity was assessed as Δ_targets − Δ_nontargets (target dimensions: active listening, personalization, and complex thinking). Performance consistency was assessed via coefficient of variation (CV), Shannon entropy across dimension profiles, and the targeting differential. Bias-corrected bootstrap confidence intervals (B=5000 iterations) were computed for all effect size estimates. All P values are reported as exact values per Journal of Medical Internet Research (JMIR) guidelines.

The prespecified dual-criterion exclusion rule required both statistical extremity (|z|>2.5 SD on the participant-level overall mean) and qualitative evidence of task invalidity. Participant P16 satisfied both criteria: the Version B participant-level mean (1.29) produced a standardized score of z=−2.60, and open-ended feedback indicated that the participant was evaluating the general training characteristics of the underlying language model rather than the chatbot interaction itself, rendering the data substantively invalid rather than merely variable. As a sensitivity analysis, the primary within-subject effect size was recomputed including P16 (n=16): the mean gain was approximately 1.64 points and the effect size was dz=0.49 (small-to-medium), compared with the prespecified analytic result of dz=0.881 (large) for the N=15 cohort. The primary analysis is retained as prespecified; the sensitivity result is reported in the interest of transparency.

One retained participant (P5) assigned identical scores of 5 to all 7 dimensions for version B, consistent with central tendency bias — a well-documented psychometric phenomenon reflecting a valid but conservative rating style. This pattern is qualitatively distinct from the P16 exclusion criterion, which was based on convergent statistical extremity and direct qualitative evidence of task misunderstanding. Retention of P5 is conservative with respect to the primary comparison, as uniform midpoint ratings reduce rather than inflate the reported mean gain.

The analytic cohort comprised 15 participants (demographic characteristics are reported in the Methods section). All analyses followed the prespecified within-subject analysis plan described in the Methods section, emphasizing effect size estimation and consistency metrics. Each participant evaluated both the baseline system (version A) and the enhanced system (version B) across seven therapeutic dimensions using validated 10-point Likert scales. Bias-corrected bootstrap CIs (B=5000 iterations) are reported alongside all effect size estimates.

Participants accessed both chatbot versions independently via an online platform; presentation order was self-determined and was not experimentally controlled. Accordingly, formal order effect testing was not conducted, and potential sequence effects cannot be ruled out. This represents a methodological limitation acknowledged in the Limitations section.

Mean conversational turns were 8.5 (SD 4.5, range 4‐20) for version A and 7.9 (SD 3.3, range 4‐15) for Version B, exceeding the 5‐6 turn threshold in 13 of 15 (version A) and 14 of 15 (version B) participants. Two participants in version A completed 4 turns, representing a minor deviation from the protocol threshold. Of the 15 participants, 8 (53.3%) engaged with exactly two scenarios across both versions combined, 1 (6.7%) engaged with all three scenarios, and 6 (40%) engaged primarily with one scenario per version, representing a deviation from the stated minimum of 2 scenarios per version; this is acknowledged as a protocol limitation in the Limitations section. Dimensional analyses were conducted across all available scenario–version pairings.

The enhanced system (version B) consistently outperformed the baseline system (version A) across all seven therapeutic dimensions. The overall mean therapeutic communication score (TCS) increased from 5.40 (SD 2.11) for Version A to 7.63 (SD 1.31) for Version B, representing a mean gain of +2.23 points (41%) with a within-subject effect size of dz=0.881 (95% bootstrap CI [0.32-2.20]). The bootstrap CI is deliberately wide, with the lower bound crossing into small-effect territory, reflecting appropriate uncertainty for a pilot cohort of N=15; readers should interpret effect size estimates as exploratory rather than confirmatory.

Enhancement effects were largest in the three prespecified target dimensions identified through Stage 1 expert diagnosis. Large effect improvements (targeted dimensions) were observed for Active Listening and Appropriate Questions (+3.20 points; dz=1, 95% CI [0.42-2.26]), Personalization (+2.93 points; dz=1.08, 95% CI [0.51-2.44]), and Complex Thinking and Emotions (+3.00 points; dz=0.96, 95% CI [0.42-2.16]). Medium effect improvements (secondary dimensions) were observed for empathy (+2.00; dz=0.66, 95% CI [0.17-1.46]), accuracy and usefulness (+1.87; dz=0.65, 95% CI [0.14-1.62]), and professionalism (+1.54; dz=0.55, 95% CI [0.06-1.25]). A small effect improvement was observed for the nontargeted dimension of positivity and support (+1.07; dz=0.39, 95% CI [−0.09 to 0.95]). The 95% bootstrap CI for this dimension (−0.09 to 0.95) includes zero, and this result does not exclude a null effect; the improvement should accordingly be interpreted as preliminary rather than definitive. Full dimensional comparisons are presented in Table 4.

The targeting differential — mean improvement in prespecified target dimensions (+3.04) minus nontargeted dimensions (+1.62) — was+ 1.42 points, confirming that enhancement was concentrated in clinically intended competency areas rather than distributed uniformly across all dimensions.

Individual trajectory analysis showed that 13 of 15 participants (86.7%) demonstrated improvement from Version A to Version B, with gains ranging from +0.43 to+5.86 points. Two participants showed declines (−3.86 and −1.29 points). The larger decline (−3.86) occurred in the participant with the highest baseline score (version A=8.86), consistent with a ceiling or contrast effect rather than system deterioration. The cohort-level mean improvement was Δ =+2.23 points (dz=0.881, 95% CI [0.32-2.20]), representing a large within-subject effect (Figure 4, Panel a).

Behavioral analysis of conversational logs revealed three concurrent changes in the enhanced system: average character count decreased by 44.9% (from 381 to 210 characters per response); sentence count decreased by 50.5% (from 10.9 to 5.4 sentences per response); and the proportion of question-based responses increased from 42.3% to 69.7% (+27.4 percentage points) (Figure 4, Panels b–d).

These three changes co-occurred as a result of the same enhancement protocol and cannot be fully disentangled statistically within the current pilot design. Observed rating improvements may reflect the increase in interactive questioning, the reduction in cognitive load from shorter responses, or an interaction of both. We treat the interpretation that questioning rather than length reduction is the primary driver of improvement as a hypothesis to be tested in future experimental designs with orthogonally manipulated conditions, rather than a confirmed finding of this study.

Scenario-level disaggregation was not prespecified, and full stratified reanalysis is beyond the scope of this pilot study. However, descriptive performance for the crisis scenario (scenario 4: suicidal ideation) is noted: the expert evaluator assigned the highest possible scores to Version B across all 7 dimensions in this scenario (total score 21/21), compared with 14/21 for version A, indicating that the enhanced system performed strongly on the highest-severity scenario. This descriptive finding should be interpreted cautiously given single-expert evaluation and a single scenario instance; it does not preclude the possibility that composite averaging across severity levels may mask scenario-specific variation in other subgroups. This limitation is acknowledged in the Limitations section.

Open-ended feedback from all 15 participants revealed three primary themes consistent with the quantitative improvements. Response structure: participants described baseline responses as “too long and metaphorical,” whereas enhanced responses were characterized as “short, clear, and question-driven.” Tone calibration: baseline responses were perceived as “light and dismissive” in high-severity situations, while the enhanced system was described as “serious when needed, yet supportive.” Contextual understanding: the baseline system was described as providing “the same suggestions regardless of context,” while the enhanced system felt “more human-like, like talking to a friend who understands.”

These themes align directly with the largest quantitative gains in active listening (+3.20), complex thinking (+3), and personalization (+2.93), providing convergent qualitative–quantitative evidence for the enhancement mechanisms. One critical qualitative pattern also emerged: several participants noted that the enhanced system tended to repeat a structure of asking a question followed by a longer explanation, suggesting that the mandatory follow-up question rule may have replaced structural rigidity with a new perceivable cadence. This finding and its implications for future design iterations are discussed in the Discussion section. Full thematic synthesis and representative quotations are provided in Table S1 in Multimedia Appendix 5.

A licensed clinical psychologist (PhD, 30+ y of clinical experience) conducted independent evaluation of matched transcript pairs across four mental health scenarios using a 3-point evidence-based scale (1=Poor; 2=Adequate; 3=Excellent). The evaluator was blinded to version assignment, participant identities, and scenario order.

The enhanced system was preferred in three of four scenarios (75% preference rate); the academic stress scenario (S2) favored the baseline version. Overall expert scores were 76 points (version B) versus 65 points (version A) across all scenarios and dimensions combined (Table 5). Given that a single expert evaluator was used, interrater reliability for this component cannot be estimated; expert preference findings should be interpreted as preliminary and hypothesis-generating rather than confirmatory.

Notably, the expert assigned high scores to the enhanced system’s crisis response, while participant feedback characterized the same response as professional but insufficiently warm. This divergence between expert-rated clinical appropriateness and user-experienced engagement reflects a clinically meaningful tension between adherence to safety protocols and perceived conversational warmth, discussed further in the Discussion section. Representative response comparisons for the presentation anxiety and crisis scenarios are provided in Figures S1 and S2 in Multimedia Appendix 5 .

Multistakeholder assessment integrated three evaluator perspectives: participants (86.7% preference, 13/15), expert clinician (75% preference, 3/4 scenarios), and multi-LLM evaluation using GPT 4.0 (OpenAI), Claude 4.0 Sonnet (Anthropic), and Gemini 2.5 Flash (Google). All three evaluator categories showed consistent preference for the enhanced system, with convergence ranging from 75% to 86.7%.

LLM evaluators consistently favored Version B but frequently added qualified preferences, noting that enhancement effects were context-dependent, and declined to make definitive clinical recommendations that human evaluators provided more readily. While multi-LLM evaluation surfaced recurring structural issues in the baseline system — including overuse of metaphors and insufficient crisis inquiry — it provided fewer novel diagnostic insights than the expert clinical review. This pattern underscores that LLM evaluation functions most effectively as a scalable complementary validator rather than a primary clinical judge. Full cross-evaluator comparison is provided in Table S2 in Multimedia Appendix 5.

Three prespecified consistency indicators confirmed that EvaluationPlus produced reliable and systematically targeted improvements across the clinical validation cohort (Table 6).

Within-subject residual uncertainty (SDΔ=2.53) reflects moderate variability in individual improvement magnitude, expected given the complexity of therapeutic communication assessment in a heterogeneous pilot cohort. The corresponding overall effect size of dz=0.881 (95% bootstrap CI [0.32-2.20]) indicates that mean improvement was substantial relative to this variability.

Cross-participant rating consistency improved substantially. The coefficient of variation across the seven therapeutic dimensions decreased from 20% (version A) to 8.1% (version B), indicating that participant ratings converged markedly around the group mean in the enhanced system. Mean within-dimension SD similarly decreased from 2.55 to 1.65, reflecting more uniform participant agreement on enhanced system quality.

. Shannon entropy computed from normalized per-dimension mean score profiles increased from H=1.929 (version A) to H=1.943 (version B), approaching the theoretical maximum of H=1.946 for seven equally-weighted dimensions. This increase indicates that the enhanced system produced a more evenly distributed competency profile, reducing the concentration of performance in dominant dimensions that characterized the baseline system.

Targeting precision was confirmed by a differential of +1.42 points between prespecified target dimensions (mean gain =+3.04, SD 0.14) and nontargeted dimensions (mean gain = +1.62, SD 0.41), demonstrating that the expert-guided enhancement protocol produced gains concentrated in clinically intended areas rather than global undifferentiated improvement.

This pilot study demonstrates that EvaluationPlus provides a structured, reproducible quality assurance framework for therapeutic AI systems at the feasibility-validation stage. Unlike traditional evaluation workflows that conclude with diagnostic reporting, EvaluationPlus operationalizes a 3-stage improvement loop — expert-guided diagnosis, multi-LLM gap–solution mapping, and within-subject validation — to support governed, clinically supervised enhancement. The substantial overall improvement (+41%; dz=0.881, 95% CI [0.32-2.20]) and large targeted gains in expert-prioritized dimensions confirm that structured quality-improvement protocols can selectively strengthen communication competencies without compromising nontargeted dimensions. Beyond average performance gains, improvements in cross-participant rating consistency (CV: 20% → 8.1%), within-dimension score variability (mean SD: 2.55 → 1.65), and cross-dimensional balance (Shannon entropy: H=1.929 → 1.943) indicate that the framework promotes more stable and reliable therapeutic behavior. Together, these findings establish a proof of concept for evaluation-driven enhancement aligned with health care quality assurance principles, while acknowledging that generalization to clinical populations and real-world contexts requires further validation.

Process-level analyses revealed coherent mechanisms through which the framework enhanced therapeutic quality. Strategic questioning, mandating at least one context-relevant follow-up question per response, increased question-led turns by 27.4 percentage points and was associated with improvements in active listening and personalization ratings. Severity-appropriate tone calibration reduced incongruent metaphors and over-validation in high-severity scenarios while preserving warmth in lower-severity interactions. Contextual personalization, through reflective summaries, use of prior-turn information, and reduction of generic advice, addressed baseline uniformity and yielded more “human-like” interactions reported by participants. These mechanisms align with counseling psychology evidence emphasizing focused inquiry, collaborative agenda-setting, and succinct communication [19-21].

However, an important interpretive caveat applies: the three enhancement components co-occurred within the same protocol cycles and cannot be statistically disentangled in the current design. Specifically, response length decreased by 44.9% concurrently with the increase in questioning behavior, and improved ratings may reflect reduced cognitive load from shorter responses, higher-quality questioning, or an interaction of both. Future studies should orthogonally manipulate these variables to isolate their independent contributions to therapeutic quality.

The findings clarify both the potential and the boundaries of LLM-mediated therapeutic support. EvaluationPlus demonstrates that structured, expert-guided enhancement can produce meaningful communication improvements within a pilot validation context. The multi-LLM gap–solution mapping component offers a scalable mechanism for translating expert diagnostic findings into prescriptive enhancement strategies, reducing reliance on informal prompt adjustment and supporting documentation of quality improvement decisions.

At the same time, this study assesses communication quality as rated by healthy university students, not clinical outcomes such as symptom reduction or engagement retention. These findings should not be interpreted as evidence of therapeutic efficacy. Any organizational adoption of the EvaluationPlus framework should be understood as a quality assurance methodology for iterative system improvement, not as a clinical effectiveness intervention.

The crisis scenario findings warrant particular attention. Although the enhanced system improved questioning and tone in simulated crisis interactions, it remained limited to generic referrals and could not conduct comprehensive risk assessments or individualized safety planning. LLMs lack legal accountability, lived empathy, and dynamic situational awareness essential for handling imminent risk. Crisis responses were evaluated only by healthy participants using simulated scenarios, which substantially limits the ecological validity of crisis-related findings. Accordingly, any integration of AI-based therapeutic systems in contexts where crisis presentations are possible must include hybrid safety workflows: automatic risk-cue detection, immediate human handoff options, clinician review of flagged transcripts, and monitoring of near-miss events. Institutions should codify these safeguards within governance protocols and integrate escalation pathways into electronic health record or customer relationship management systems.

Two methodological issues merit explicit discussion. First, the EvaluationPlus framework uses the same 7-dimension therapeutic competency rubric across all three stages: as the diagnostic instrument in stage 1, as the target specification in stage 2, and as the outcome measure in stage 3. This design, while intentional for methodological consistency, introduces a circularity risk: improvements measured by the rubric may partly reflect optimization toward the rubric’s own criteria rather than broader, context-independent therapeutic quality. Future validation studies should include outcome measures independent of the rubric used to guide enhancement, such as standardized therapeutic alliance scales or blinded third-party clinical assessments.

Second, participant feedback identified a new conversational pattern in the enhanced system: a perceivable rhythm of question followed by extended explanation, suggesting that the mandatory follow-up question rule may have replaced one form of structural rigidity with another. This finding illustrates a general design challenge in rule-based enhancement of conversational AI: explicit behavioral rules improve targeted metrics but may introduce secondary rigidity. Future enhancement cycles should explore probabilistic rather than mandatory questioning rules, and evaluate the naturalness of dialogue as an independent outcome dimension.

Third, the same LLM platforms used for stage 2 gap–solution mapping were also used as 1 of 3 evaluator perspectives in stage 3 multistakeholder assessment. Although the prompts were sufficiently distinct — stage 2 prompts requested prescriptive enhancement recommendations, while stage 3 prompts requested rubric-aligned evaluation of transcripts — this overlap represents a potential evaluator–mapper conflict. Future studies should use independent LLM platforms for mapping and evaluation to eliminate this circularity concern.

This pilot study demonstrates the feasibility of EvaluationPlus as a structured, reproducible framework for quality assurance in therapeutic AI systems. By integrating expert-guided diagnosis, multi-LLM gap–solution mapping, and within-subject validation, the framework produced substantial and targeted improvements in therapeutic communication quality (overall dz=0.881), with consistent preference from both user (86.7%) and expert evaluators (75%). Methodological contributions include the operationalization of the evaluation-to-enhancement loop, the use of multi-LLM triangulation for prescriptive gap mapping, and the development of consistency metrics appropriate for pilot-stage validation. Although the study is limited by sample size, cultural specificity, single expert evaluation, and simulated crisis scenarios, it establishes a clear proof-of-concept for governed, evaluation-driven quality improvement in therapeutic AI development. Future work extending validation to diverse populations, longitudinal clinical outcomes, and hybrid crisis-response protocols will be essential for establishing the external validity and clinical use of this approach.