In this study, we selected 3 models from 3 widely used LLM platforms for evaluation and comparison. Specifically, we evaluated ChatGPT with GPT-5.2-thinking (with extended thinking; OpenAI), Opus (version 4.6; Anthropic), and Gemini (Gemini 3 Pro; Google) [31-33]. During the study window, extended thinking modes were available for ChatGPT and Opus but not for Gemini; therefore, each platform was evaluated using the relevant configuration available through its consumer-facing web interface at the time of analysis.

We intentionally used the consumer-facing web interfaces rather than the corresponding application programming interfaces (APIs), and each model-workflow condition was executed once using a standardized prompt and independent conversation, because the goal of this study was to evaluate a practical, researcher-facing workflow under conditions that resemble how many qualitative researchers currently interact with these tools in applied settings, rather than to benchmark API-level performance under fully programmatic control. This choice also allowed direct assessment of operational factors relevant to multistep qualitative analysis, including conversation organization, output exportability, and adherence to structured prompts within the native user interfaces.

All data collection and model interactions were conducted within a fixed study window from February 10, 2026, to March 30, 2026, to provide a temporal anchor for the model versions available at the time of evaluation. To reduce variability related to prior interaction history, all analyses were performed using newly registered accounts with paid personal subscriptions (ChatGPT Plus, Claude Pro, and Gemini Pro). Where available, settings that allow models to reference prior conversations were disabled (eg, “Reference chat history” in ChatGPT). Each analytical subtask was performed in a new, independent conversation so that no model could rely on prior session memory. The same standardized prompts, workflow logic, and transcript inputs were applied across all 3 platforms.

To conduct thematic analysis, we applied a human-AI collaboration framework developed in our prior study [30]. In this framework, the overall analytical workflow is decomposed into a series of discrete, model-specific subtasks, each executed by a specialized LLM module. Each subtask is performed within an independent LLM conversation, and modules do not share memory or contextual knowledge across sessions. This modular architecture reduces per-module complexity, mitigates data loss caused by context-window limitations, lowers computational overhead, and improves the consistency and quality of the generated outputs. The framework consists of 3 specialized modules: a code extraction module, a code combination module, and a theme generation module. Figure 1 shows the prompts for the 3 modules.

The code extraction module is responsible for identifying all relevant qualitative codes from an input transcript. A standardized prompt is provided alongside the transcript, instructing the model to identify as many relevant codes as possible, assign each code a concise name limited to 10 words, and produce an approximately 80-word description for each code. The prompt also requires the model to cite the corresponding interview question number as supporting evidence, use clear and concrete language, avoid clichés or vague generalizations, and generate outputs in a predefined standardized format.

The code combination module consolidates overlapping or redundant codes in order to produce a more coherent and streamlined codebook. Using another standardized prompt, the model is instructed to group codes with substantially similar titles and descriptions and generate a new name and definition for each merged group. For codes without close counterparts, the original title and description are retained. To ensure traceability throughout the analysis process, the module records the original code number or numbers associated with each merged group, and the final results are produced in the same prescribed output format.

The theme generation module derives higher-level themes from the consolidated codebook. Its prompt directs the model to cluster codes and their descriptions into themes that reflect conventions commonly used in human-authored qualitative research. Detailed guidelines regarding length constraints, thematic structure, and stylistic requirements are embedded in the prompt to guide the generation process. Each code must be assigned to exactly 1 theme, ensuring complete coverage of the dataset without overlapping. The final output consists of a comprehensive list of themes, with each theme accompanied by a title, a descriptive summary, and the original code numbers associated with the codes it subsumes.

Figure 2 illustrates the workflow of the human-AI collaboration framework. The input for our framework would be a set of transcripts of interviews. Each transcript was uploaded as a .txt file into a new LLM conversation together with the standardized prompt for the code extraction module. Initiating a separate conversation for each transcript ensured that the interview context was preserved while respecting token-length constraints, maintained uniform analytical conditions, and prevented contamination from prior interactions.

After code extraction, one researcher (YB) collected the model-generated codes in text files. To reduce manual effort and the risk of identifier transcription errors, a short Python script (Python Software Foundation) was used solely to assign unique identifiers to the collected codes and generate 2 structured text files: a layer 1 (L1) codebook containing the code number, name, and definition for each code, and a quote-trace file containing the code number, code name, and corresponding interview question numbers. This deterministic organizational step did not modify code content, conduct qualitative analysis, or involve programmatic interaction with the LLM platforms, and it could be performed manually if needed.

To reduce redundancy, the code combination module was applied. The L1 codebook was divided into 4 subcodebooks to control the input length for each LLM conversation; each subcodebook contained the codes and descriptions derived from 4 to 5 transcripts. This partition was selected as a practical heuristic to balance input-length constraints and codebook diversity. Specifically, grouping 4-5 transcripts ensured that each subcodebook remained within a manageable size for the consumer-facing web interfaces used in this study, while still preserving sufficient variation in codes to support meaningful consolidation at the L2 stage. This approach allowed iterative merging without exceeding response length limits or overly compressing heterogeneous concepts in a single step.

These 4 subcodebooks were submitted independently to new LLM conversations together with the standardized code combination module prompt. The resulting outputs were then collected and processed to form the layer 2 (L2) codebook. To further consolidate overlapping codes, the L2 codebook was submitted again to the code combination module, producing the layer 3 (L3) codebook. Throughout this iterative consolidation process, parent-layer code numbers were preserved to maintain traceability between newly generated codes and their original source quotes.

In addition to this hierarchical workflow, we evaluated 2 straightforward strategies that did not impose input length controls. In the first approach, all transcripts were merged into a single txt file and submitted directly to the code extraction module, producing a direct coding (DC) codebook. In the second approach, the L1 codebook was submitted directly to the code combination module without dividing it into subcodebooks, producing a direct grouping (DG) codebook. Finally, all 5 resulting codebooks—L1, L2, L3, DG, and DC—were independently submitted to the theme generation module in separate LLM conversations using the same standardized prompt. The themes generated were then used for subsequent evaluation and comparison across the different analytical strategies.

The semistructured qualitative interviews were collected as part of 3 usability studies using a mixed methods design [27-29]. The studies aimed at assessing usability, attitudes, and acceptance of home-based telerehabilitation systems in patients with 1 of 3 conditions: COPD, POTS, or related dysautonomia, and ILD. All study participants were aged 18 years or older. They also needed the ability to complete study procedures, including usability testing and a semistructured interview. Individuals were excluded if they had medical conditions that could increase risk during the exercise‑related usability evaluation, including unstable angina, uncontrolled hypertension, a recent myocardial infarction, pacemaker use, painful or unstable bony metastases, recent skeletal fractures, or any other condition that, in the clinical or research team’s judgment, would make participation unsafe. Enrolled participants attended a single, 1‑hour, in‑person visit. After providing informed consent, research staff demonstrated the telerehabilitation system and supervised a practice session and then administered a sociodemographic questionnaire. Usability testing comprised 3 standardized tasks completed independently: logging into the system, completing a survey, and reviewing and performing an exercise routine. Analyses were stratified by the underlying medical conditions.

After usability testing, participants took part in a semistructured interview with open‑ended questions to explore their perceptions of the system and its potential for home‑based symptom management. Interviews were transcribed verbatim and deidentified. The qualitative interview data were analyzed using thematic analysis following Braun and Clarke’s [1] 6-phase process to organize, identify, and interpret key patterns and themes. The qualitative analysis team comprised a qualitative researcher (Aileen Gabriel) and a clinician-scientist (JF). The disagreements were resolved by a clinical informatician familiar with qualitative analysis. The iterative analysis continued until all codes were finalized and disagreements were reconciled. The research team repeatedly reviewed transcripts to immerse themselves in the data; initial codes were generated inductively through open coding, iteratively refined, and grouped into broader categories to capture patterns across participants. These categories were then synthesized into themes representing key aspects of participant experiences [27-29].

For evaluation, we used the original deidentified interview transcripts and the final human-generated thematic analysis results from the 3 source studies. The ILD dataset included 17 transcripts, for which the human analysis generated 169 codes and 7 themes. The POTS dataset included 15 transcripts, for which the human analysis generated 178 codes and 10 themes. The COPD dataset included 16 transcripts, for which the human analysis generated 261 codes and 6 themes. The original transcripts were submitted as input to the LLM-assisted workflow, and the corresponding human-generated themes served as the reference standard for evaluating framework-derived themes.

All interview transcripts used in this study were deidentified and were not publicly available, published, or released before the present analysis. Therefore, direct exposure of the evaluated models to the source transcripts through publicly accessible training data is unlikely. The code extraction module prompt also included a study background field that was populated separately for each disease cohort. The exact verbatim text used for the ILD, POTS, and COPD datasets is provided in Multimedia Appendix 1.

To quantitatively assess the similarity between human-generated themes and those produced by the LLM, we used the sentence-transformer model sentence-t5-xxl, which has demonstrated strong performance on sentence similarity tasks, to embed the theme texts into a 768-dimensional dense vector space [34]. Because this model has a bounded maximum input length, we verified that all theme texts were below this limit; therefore, no truncation was applied during embedding. The resulting vectors were normalized to remove length bias and ensure that comparisons reflected semantic similarity rather than differences in text magnitude. Cosine similarity was then calculated for each pair of human and LLM theme vectors. Cosine similarity scores range from −1 to +1 and indicate the degree to which 2 texts describe conceptually similar content, with higher values representing greater semantic similarity. In addition, to verify traceability fidelity, we developed a Python script to validate whether each propagated parent-layer code identifier referenced an existing valid code in the immediately preceding codebook layer. The number of themes identified by humans and by the LLM often differed. To compare unequal theme sets while preserving both alignment quality and complete theme coverage, we used a 2-stage matching procedure combining the Hungarian algorithm and a Greedy nearest-neighbor assignment [35,36]. Figure 3 illustrates this alignment process.

First, the Hungarian algorithm was applied to the cosine similarity matrix to identify the globally optimal one-to-one alignment between the smaller theme set and an equally sized subset of the larger theme set, thereby establishing the primary bidirectional correspondences. Using the Hungarian algorithm alone would have excluded the remaining unmatched themes in the larger set from evaluation, which could bias the final similarity upward by omitting surplus themes with lower semantic similarity. Therefore, any remaining unmatched themes in the larger set were subsequently assigned to their most semantically similar theme in the smaller set using a Greedy nearest-neighbor procedure, ensuring that all themes contributed to the evaluation rather than discarding surplus themes. A Greedy-only strategy was not used because it does not enforce a globally optimal one-to-one primary alignment, and some themes in the opposite set may remain without any primary correspondence.

When multiple themes from the larger set mapped to the same theme in the smaller set, the cosine similarity values for those mappings were averaged rather than retaining only the single highest-scoring match. As a result, generating additional themes did not inherently improve the final score; extra weakly aligned themes could reduce the aggregate similarity. Basic statistical measures (average, standard deviation, maximum, median, and minimum) were then calculated for the resulting cosine similarity scores.

Processing time was measured manually using a digital clock or stopwatch. Timing began when the prompt and input file were submitted through the web interface and ended when the model completed its full response. Therefore, the recorded time represents end-user wall-clock processing time, including web interface latency, model response generation, and streaming time, rather than isolated model inference time. For theme generation tasks, processing time was measured 3 times, once for each study dataset. For L1 code extraction, 5 transcripts were randomly selected from the 48 available transcripts, and the corresponding extraction tasks were timed for each model. For the other tasks, processing time was measured across 5 randomly selected observations. Reported values are presented as mean (SD) in seconds.

We also conducted a brief descriptive assessment of output format consistency and workflow management. Format deviations were defined as responses that did not follow the prompt-specified structure and required manual correction before downstream processing, such as missing required fields, inconsistent numbering, omitted source or parent-code identifiers, or unrequested formatting changes. One researcher (YB) reviewed the generated outputs and recorded distinct format patterns. Workflow management was assessed descriptively based on whether the web interface supported project-level organization, easy tracking of multiple conversations, and convenient copying or exporting of outputs. Interrater agreement was not assessed because this was a practical workflow observation rather than a formal usability evaluation.

This study did not involve human participants or the collection of identifiable private information. No patient identifiers were accessed, recorded, or analyzed, and no attempt was made to reidentify individuals. In accordance with JMIR guidelines, this research does not meet the definition of human subjects research and was therefore exempt from institutional review board review and informed consent requirements. The study was conducted in compliance with applicable ethical standards for research using deidentified secondary data. The qualitative interview protocol was approved by the institutional review board of the University of Utah (protocol IRB_00168937).

This study evaluated a modular human–AI collaboration pipeline for LLM-assisted thematic analysis across 3 qualitative digital health interview datasets and 5 workflow strategies. Overall, the findings show that both model choice and workflow structure shaped thematic outputs, but model choice appeared to have a stronger influence on alignment with human-generated themes. Across ILD, POTS, and COPD, Opus produced the most consistently human-aligned themes, while ChatGPT performed competitively in selected settings, and Gemini showed more constrained but relatively stable performance. These findings suggest that LLM-assisted thematic analysis should not be treated as a single generic capability; rather, performance depends on how a model interacts with specific workflow designs for extraction, consolidation, and theme generation.

A key finding is that Gemini and ChatGPT exhibited constrained output behavior under the evaluated conditions. Both models produced relatively fixed numbers of codes across tasks (approximately 10 for Gemini and approximately 25 for ChatGPT), even when input size and task complexity varied. In qualitative analysis, this matters because thematic fidelity depends on adequately capturing the diversity of participant experiences. When outputs are effectively bounded, there is a risk of undercoding distinct ideas and underrepresenting less frequent but potentially important perspectives. This risk is particularly important in health-related interviews, where less frequent concerns may still be clinically meaningful [18-20].

The word count analysis provides additional evidence supporting this pattern. Despite substantial variation in input length across workflow stages, Gemini and ChatGPT produced outputs with relatively stable lengths, suggesting the presence of an effective output ceiling under the evaluated conditions. In contrast, Opus exhibited more input-responsive behavior, with output length increasing in higher-input settings such as DC and large codebook consolidation. This difference in output length responsiveness is consistent with the observed variation in code and theme counts across models.

At the same time, the findings show that workflow structure can partly compensate for model-specific limitations. For Opus, the hierarchical strategies performed especially well, with L3 producing the highest or near-highest similarity in 2 of the 3 studies, and strong performance was also observed in DC and DG settings. This pattern suggests that Opus was able to absorb longer inputs and iterative consolidation without substantial thematic drift. In practical terms, this is encouraging for larger qualitative projects: a staged workflow that first extracts detailed codes and then merges them iteratively may reduce redundancy while still preserving conceptual distinctions. The strong performance of DC and DG for Opus also indicates that the model can tolerate broader inputs, giving researchers flexibility to choose between hierarchical and direct strategies depending on dataset size, desired granularity, and time constraints.

For ChatGPT, a different pattern emerged. Its performance was relatively stronger in L1 and L2, with less consistent results after deeper consolidation. This suggests that ChatGPT may benefit from moderately structured intermediate representations but lose details when codes are repeatedly merged. One interpretation is that each consolidation step reduces input richness, and because ChatGPT also showed capped output behavior, repeated merging may amplify information loss. In this sense, ChatGPT appears well suited to workflows that preserve more of the original code diversity rather than heavily compressing it. For small- to medium-sized datasets, this may make L1 or L2 especially useful, whereas for larger datasets requiring multiple rounds of consolidation, the trade-off between manageability and thematic detail may become more pronounced.

Gemini showed the lowest overall similarity scores in most settings, but its results were not uniformly weak. In several direct-coding settings, it remained competitive, and its performance was relatively stable across studies. However, Gemini’s strong tendency to generate approximately the same number of codes regardless of input size suggests limited responsiveness to transcript complexity. This likely constrained downstream theme generation by reducing the range of available concepts before theming even began. From a practical perspective, Gemini may still be useful when the analytic goal is rapid, high-level summarization rather than detailed qualitative interpretation. Its short outputs and fast runtimes make it efficient, but that efficiency appears to come at the cost of granularity and formatting consistency.

The runtime findings reinforce this interpretation of model behavior. Across the pipeline, processing time was driven less by the nominal task type than by cumulative input and output length. Opus had the longest runtimes because it also generated the most extensive output, which then increased the size of the next-stage inputs. Gemini was consistently the fastest, reflecting its shorter and more compressed responses. This highlights an important trade-off in applied LLM-assisted qualitative research: the most detailed outputs may better preserve thematic nuance, but they also require more time and create heavier downstream processing burdens. Therefore, researchers need to balance fidelity, scalability, and efficiency when selecting a platform and workflow.

Another practical contribution of this study is that it evaluates not only thematic similarity but also the operational usability of the platforms for multistep qualitative work. ChatGPT and Opus more reliably followed the required output structure and supported easier management of multiple conversations through project-level organization. Gemini was less consistent in output format, which increased preprocessing effort. These differences may seem secondary to thematic quality, but they matter in real-world research settings. A workflow that requires dozens of extractions, combinations, and theme generation sessions depends on consistent formatting and manageable conversation organization. Thus, platform usability should be considered alongside analytic performance when choosing tools for LLM-assisted qualitative analysis.

Compared with prior literature, this study contributes a more explicitly workflow-centered evaluation. Recent studies have shown that LLMs can support qualitative coding, content analysis, grounded theory, and thematic summarization, including comparisons between AI-generated and human-generated qualitative analyses [13-17,23-25]. However, many prior studies rely on single-prompt, single-model, or near end-to-end approaches. These approaches can be efficient, but they make it harder to identify where information loss occurs and often provide less traceability across analytic stages. By contrast, our pipeline separates code extraction, code consolidation, and theme generation into independent, standardized steps. This structure makes the analytic process more auditable and allows direct comparison of hierarchical versus direct strategies, aligning with recent recommendations that LLM-supported qualitative analysis should emphasize transparency, reproducibility, validation, and human oversight [8,10-12,20,22,24-26].

This study also adds a reproducible quantitative evaluation layer by comparing AI-generated themes with human-generated themes through embedding-based semantic similarity and systematic alignment. Although semantic similarity cannot fully capture qualitative rigor, it provides a useful comparative signal across models, workflows, and datasets [6,8,20]. In this study, the use of 3 separate health interview datasets strengthens confidence that the observed patterns are not tied to a single sample alone. Across ILD, POTS, and COPD, Opus remained consistently strong, while the relative strengths and weaknesses of ChatGPT and Gemini followed broadly similar patterns. This cross-study consistency suggests that workflow-model interactions may be a meaningful area for future methodological research in LLM-assisted qualitative analysis.

Several limitations should be noted. First, all analyses were conducted through consumer-facing web interfaces rather than APIs. Although we used newly registered paid accounts, disabled memory settings where available, and restricted data collection to a fixed study window, web-based platforms may still be affected by unannounced model updates, interface-level output constraints, and uncontrolled generation settings. In addition, the available configurations differed across platforms: ChatGPT and Opus were evaluated with extended thinking modes, whereas Gemini did not offer a comparable extended thinking option during the study window and was evaluated in its available standard configuration. Accordingly, Gemini’s shorter outputs and generally lower similarity scores may reflect differences in the available model-interface configuration, in addition to differences in model behavior.

Second, the study used standardized prompts and a modular workflow, which improved consistency across models but may not reflect performance under alternative prompt designs or less structured qualitative analysis workflows. In addition, each model-workflow condition was executed once to mirror practical researcher-facing use, so within-model stochastic variation was not quantified.

Third, the evaluation relied on embedding-based cosine similarity and a 2-stage Hungarian plus Greedy alignment procedure. This approach allowed all themes to be included in the comparison, but it is not equivalent to a formal precision-recall framework and does not fully capture qualitative rigor, interpretive depth, or potential overgeneration effects.

Fourth, this study was descriptive rather than confirmatory. Formal inferential testing was not conducted because theme pair similarity scores were generated through an alignment procedure, were not fully independent, and came from only 3 datasets with 1 run per model-workflow condition. Therefore, differences across models and strategies should be interpreted as descriptive patterns rather than statistically confirmed effects.

Although all LLM interactions were performed through consumer-facing web interfaces, to reduce manual labor and the risk of human error, this implementation used a simple Python-assisted identifier assignment step to organize intermediate outputs, which requires intermediate programmatic data-processing skills.

Finally, the study evaluated only 3 LLMs and included a descriptive assessment of output format consistency and workflow management conducted by one researcher (YB). Future work should include additional proprietary and open-source models, repeated runs, independent workflow assessments, and prespecified statistical analyses across larger and more diverse qualitative datasets.

Future work should validate this framework across larger and more diverse qualitative datasets, disease areas, and methodological traditions. Additional studies should include a broader range of proprietary and open-source LLMs, repeated runs for each model-workflow condition, and API-based implementations with controlled decoding parameters to better assess reproducibility and run-to-run variability. Evaluation should also extend beyond embedding-based semantic similarity by incorporating expert ratings of theme quality, code-level validation linking excerpts to assigned codes, human-human comparison baselines, and measures of thematic coverage, missed concepts, and overgeneration. Future research could further examine how prompt design, transcript partitioning strategies, and human intervention at different stages influence analytic fidelity. These efforts would help clarify which combinations of model choice, workflow design, and human oversight are most appropriate for different qualitative research goals.

This study shows that a modular human-AI pipeline can support thematic analysis across multiple digital health interview studies, but its effectiveness depends strongly on both model choice and workflow design. Opus achieved the most consistent alignment with human-generated themes, while ChatGPT and Gemini showed distinct trade-offs in thematic fidelity, runtime, and usability. These results suggest that LLMs can serve as valuable assistants in qualitative research when embedded within structured, transparent, and human-supervised workflows. Rather than replacing qualitative researchers, LLMs may be most useful as configurable analytic partners whose strengths and limitations must be carefully matched to the goals and scale of the study.