Concept mapping (CM) is a structured research method designed to visualize and organize thoughts, ideas, or knowledge concerning a specific topic by creating graphical representations of concepts and their interrelationships [1]. Widely used in disciplines such as education, psychology, health sciences, and other social sciences, CM is an integrative mixed methods research methodology for exploring and analyzing complex conceptual domains. It includes both qualitative and quantitative data collection and analysis in sequential steps, where each step builds on the previous one, as described below [2]. One of the strengths of this method is the active involvement of stakeholders throughout the process. Depending on the project and the research question, the participants can be more or less involved in all stages described in the following sections [3].

After the initial preparation stage, the CM methodology typically comprises the following sequential stages, as shown in Figure 1. Normally, idea synthesis is part of idea generation, but for clarity in this study, it is defined here as an additional step in the process.

The idea synthesis phase in CM has traditionally relied on manual, labor-intensive procedures conducted by research teams through iterative consensus-building exercises. A range of approaches has been documented in the literature. For instance, Ashe et al [7] describe a multistage process in which individual review of items was followed by several collaborative meetings to finalize the list of unique ideas. Similarly, Windsor [8] employed 5 independent coders to identify redundancies, after which consensus was reached on the items to retain. Hassmiller Lich et al [9] introduced a more advanced strategy by comparing an initial set of 830 ideas with a final curated set of 97 to assess representativeness. Pantha et al [2] conducted a review indicating that while approximately one-third of studies provided an explicit rationale for their sample size, three-quarters described their idea synthesis process in detail. However, concerns about methodological transparency remain. For example, McLinden [10] criticized the often insufficient reporting of synthesis procedures, arguing that such omissions hinder the ability to evaluate the representativeness and validity of the final idea sets. These studies underscore the variability in practice and the need for more standardized and transparent approaches to idea synthesis in CM research.

Despite growing awareness of the importance of rigorous idea synthesis, inconsistencies persist in methodological reporting and implementation. Even though digital technologies have enabled larger-scale data collection, they have also further compounded the challenges and increased the burden on researchers to conduct transparent and methodologically sound synthesis. While Kane and Trochim [1] emphasized that CM relies on a combination of technological support and human qualitative judgment—a “high-tech, high-touch” approach—there is a growing need for methods that blend automation with interpretive rigor without compromising the qualitative depth of the process.

To address all the aforementioned challenges, this study presents the development and evaluation of IdeaDistiller, a semiautomated tool designed to support the idea synthesis phase of CM. By leveraging machine learning techniques, our approach aims to reduce researchers’ burden, enhance methodological transparency, and preserve qualitative depth. We demonstrate the tool’s utility through a series of exploratory benchmarks across diverse English and Swedish datasets, providing a framework for more efficient and reproducible conceptual analysis in large-scale studies.

Recognizing that idea synthesis is inherently a qualitative task requiring nuanced judgment, we adopted a human-in-the-loop (HITL) approach, integrating automated techniques with expert oversight. This strategy aligns with the recommendation by Kane and Trochim [1] that, while technology can support the CM process, human interpretive engagement remains essential. We acknowledge that the distillation of ideas is not a discovery of an objective “ground truth” but rather a subjective interpretive process in which granularity and categorization depend on the researcher’s perspective. Overall, we aimed at developing a tool that fulfills the following requirements:

Our approach is based on the principle that all ideas can be clustered into semantically coherent groups, where each group consists of ideas conveying the same or a similar idea in different formulations, and the main idea for each group is distinct from all other groups. Creating such a clustering enables the researcher to subsequently extract one representative idea from each group, ensuring that all extracted ideas collectively represent the diversity of the complete dataset.

We explored several approaches to clustering ideas based on their semantic similarity. In the following sections, we describe the data, clustering methods, and evaluation metrics used, as well as the output created to facilitate the easy integration of human feedback.

This study used data from various independent CM studies. Each dataset comprised two components: (1) the complete set of raw ideas provided by participants and (2) the manually identified set of unique ideas, which served as the reference for evaluating the performance of the computational methods developed.

In total, 9 datasets were analyzed: 6 in English and 3 Swedish datasets. The English datasets cover diverse topics within the health and health care domains, including obesity, stress, medical services [11], drug abuse [12], and aspects of suicide screening [13]. The datasets covering the first 3 topics originally contained Spanish ideas collected from Latino participants in Cincinnati; however, their English translations were used in this study. The Swedish datasets addressed cancer rehabilitation [14], the support structures available to patients and their families (the “Kraftens hus” initiative) [15], and the use of development plans as a tool in patient care [16].

Before clustering, exact duplicate entries, which were particularly frequent among shorter ideas, were removed as they would not contribute additional unique ideas. However, within our approach, it is possible to retain duplicates to allow for a quantitative follow-up analysis.

Table 1 provides an overview of the datasets, along with their sizes and the number of manually identified unique ideas. The datasets vary in thematic focus, language, participant demographics, and dataset size, providing a robust basis for evaluating the generalizability and performance of the proposed computational approach.

Semantic similarity identifies latent patterns within textual datasets by uncovering recurring groups of words that represent underlying themes or topics. Traditional methods, such as latent Dirichlet allocation [17,18], primarily rely on word frequency statistics to discover topics, which makes them limited in their ability to find semantic similarity in sentences with very different wording. Therefore, our approach builds on the BERTopic (bidirectional encoder representations topic modeling) framework [19-21], a technique that incorporates textual relationships and semantic meaning.

BERTopic follows a flexible pipeline comprising several steps. Since our primary interest lies in grouping each idea into a semantically coherent cluster, we focus on the first three stages of the pipeline: (1) embedding the ideas into a dense vector space, (2) reducing the dimensionality of the embedding space, and (3) clustering the embedded original ideas to form semantically related groups.

The core principle of BERTopic is embedding sentences into a high-dimensional embedding space to transform textual data into dense vector representations before clustering them. Semantic similarities between texts are reflected by their proximity in the vector space. These embeddings allow for a more nuanced understanding of similarity beyond simple word counts. In this study, we used several different pretrained embedding models, adapted to different languages, to create the sentence embeddings.

Through this process, all ideas are assigned to clusters, each containing original ideas with similar semantic meanings. CM researchers can then extract one or more representative ideas from each cluster. Each of the 3 mentioned steps offers multiple methodological choices, and we selected the specific methods that showed the most robust results across our exploratory study. While this process provides a good starting point, manual refinement is still needed. To simplify this task for the researcher, we additionally sort all ideas based on their cluster assignment and the semantic similarity between the clusters; that is, we organize the clusters such that those covering similar themes are positioned closer together in the output. To achieve this, we applied hierarchical clustering to the topic representations and used the resulting dendrogram to order the output.

Standard evaluation metrics for topic modeling, such as coherence scores and perplexity, assess the semantic consistency and interpretability of the identified topics. In our case, however, the availability of manually curated unique ideas from previous CM studies enabled more targeted evaluation measures. Rather than measuring “correctness” in an absolute sense, these metrics quantify the degree of alignment between the computational output and a specific, human-led reference synthesis. To quantify the performance of IdeaDistiller, we developed 2 coverage-based metrics that assess the alignment between the computationally generated clusters (without human refinement) and the expert-identified unique ideas. These metrics are based on the heuristic that an optimal alignment with the reference standard would produce as many clusters as there are unique ideas, with 1 unique idea representing each cluster. Therefore, the metrics used consider the overlap between computational clustering and curated unique ideas, which is the number of clusters containing at least 1 unique idea. The 2 metrics are defined as follows:

where n_unique_ideas represents the total number of manually curated unique ideas. This metric evaluates the proportion of unique ideas represented by the computational clusters. In other words, this metric tells us how many of the unique ideas we would obtain if we extracted exactly 1 unique idea from each cluster and how many this computational approach would miss:

where n_clusters is the total number of computationally generated clusters. This metric assesses the proportion of clusters containing at least 1 manually identified unique idea, highlighting the extent to which the computational clusters align with expert curation. A value below 1 indicates clusters without a unique idea, which can be either due to suboptimal clustering (ie, the additional clusters do not consist of new ideas but should be a part of another cluster containing a unique idea) or due to the clustering identifying new aspects that the human evaluator has not considered.

As an example, consider a scenario with 100 clusters and 96 unique ideas, where 62 clusters contain at least 1 unique idea. In this case, the 2 metrics are calculated as follows:

This example indicates that 38% (38/100) of clusters do not contain a manually curated unique idea, while 35.41% (34/96) of unique ideas are not captured when extracting 1 idea per cluster. Together, these metrics provide a comprehensive assessment of the representativeness and completeness of the clustering results. It is important to note that a lack of perfect correspondence (values below 1) does not necessarily indicate “error” by the model. Because unique ideas in CM are interpretive and dependent on the researcher’s chosen level of granularity, discrepancies may represent valid alternative structures. Therefore, these metrics should be interpreted as a measure of how closely the IdeaDistiller mimics the specific synthesis decisions made by previous expert researchers.

While these 2 metrics aim to evaluate the results from 2 different perspectives, we applied the pipeline by choosing the number of clusters to be equal to the number of manually curated unique ideas, which resulted in identical values for both metrics. Therefore, in the following, we will report only one of the metrics.

Although coherence was not used to select the best-performing model, we computed a coherence score for each idea to support researchers during the manual evaluation phase. In our setting, coherence measures the degree of semantic similarity among all ideas within the same cluster, with higher coherence indicating greater internal consistency. This score can help assess how well an idea fits within its assigned cluster, allowing for special focus on ideas with low coherence scores during the postcomputational or refinement phase. In practice, we computed embeddings for all individual ideas using a pretrained sentence-BERT (bidirectional encoder representations from transformers) model [22]. The coherence score of each idea is then calculated as the cosine similarity between its embedding and the mean embedding of all ideas in its assigned cluster.

Our approach, IdeaDistiller, produces an output file designed to support CM researchers during the idea synthesis phase. This file includes a comprehensive table listing all participant ideas alongside the results of the computational analysis, facilitating manual review and refinement.

The output contains the following components:

These outputs offer researchers a structured foundation for conducting a comprehensive evaluation and refinement of the results to finalize the idea synthesis process.

All projects that contributed data for this work have been ethically assessed and/or approved. Further details can be found in the respective manuscripts [11-16]. For this study, permission to use the data was granted by the data collectors, and no actual outcomes are reported. The data were used exclusively for the development and testing of the tool and are neither required for its further use nor can they be retrieved from the tool.

To provide a concrete example of how coherence scores reflect the semantic unity of the output, Table 3 illustrates a cluster (ID 86) from the English dataset “Stress.” The scores represent the cosine similarity between individual statements and the cluster centroid. Higher scores (eg, 0.76) correspond to statements that align closely with the core topic, while lower scores (eg, 0.52) indicate less closely related concepts like “support.” This allows researchers to quickly validate the thematic consistency of the tool’s output and identify potential outliers.

We conducted a small audit on one of the English datasets (“Drug abuse”) to gain insights into the semantic clusters without unique ideas. We found that, in many cases, these clusters did not contain novel content but slight variations of already selected ideas. However, some of these clusters contained ideas that could be deemed “novel” depending on the focus of the study. One example is 3 statements on background checks (“give background checks when you apply for a license,” “background checks on patients,” and “ask for consensual background checks to know if a person is on drugs or has experienced…”), with none of them being included in the list of unique ideas. For the given study, these ideas were considered out of scope and at the same time loosely related to 2 retained unique ideas (“The federal government should run a program that takes convicted drug dealers/users into mandatory rehab” and “Hold addicts accountable for staying clean and away from drugs”). However, depending on the focus of the study, an idea specifically about background checks could be a useful addition. This example illustrates how scope and granularity decisions shape what is included as a unique idea and shows that the tool provides a valuable starting point for idea synthesis. However, it remains independent of these study-specific decisions and therefore does not replace researcher judgment.

This study shows that semantic clustering can be used to support the idea synthesis process in a CM study. For 9 different datasets, IdeaDistiller was able to reproduce expert synthesis with an average alignment of 64%, highlighting both the value of the tool in organizing the original ideas as well as the need for expert involvement in the curation process. The latter is facilitated by providing sorted lists of all original ideas, based on which the results can easily be adjusted.

Topic modeling, specifically BERTopic, has been successfully applied in various contexts, such as Twitter (subsequently rebranded X) tweets [20]. Our study extends these findings and demonstrates the potential to enhance the CM process, specifically the idea synthesis step, using various health care–related datasets. IdeaDistiller, our proposed approach, significantly reduces the manual effort traditionally required to identify and group similar ideas by automating the clustering and thereby presorting the participant-generated ideas. This automation enables researchers to allocate more time to the subsequent analysis and interpretation of ideas rather than the manual data processing phase.

Moreover, the approach provides additional outputs, including coherence scores and semantically sorted clusters, which facilitate a more structured and efficient review process. IdeaDistiller is designed to support CM researchers, while keeping them in control of their own analysis. Extracting final, unique ideas from each cluster still necessitates human judgment, and suggested representative ideas require expert validation, particularly in complex or sensitive domains such as health care.

Our findings align with Prescott et al [23], who argue that while generative artificial intelligence (AI) can significantly reduce time and resource demands, hybrid approaches incorporating HITL validation remain necessary. Similarly, Morgan [24] concludes that although AI tools demonstrate great promise for supporting coding tasks, integrating AI with qualitative data analysis presents challenges. In agreement with Prescott et al [23], human analysts retain a crucial advantage in identifying nuanced meanings and interpreting context-specific themes—an essential skill for rigorous qualitative research.

Integrating an automated step into the idea synthesis process offers substantial benefits for methods such as CM. These include improved scalability for handling larger datasets. We also concur with Chen et al [25], who advocate for using machine learning to identify inconsistencies and ambiguities in qualitative coding. By automating the simpler stages of coding, researchers can allocate their expertise more efficiently to the complex interpretative tasks that require theoretical sensitivity and contextual knowledge—an outcome facilitated by the tool developed in this study.

A limitation encountered in this study was the variability in dataset sizes and languages, which posed challenges for optimal parameter tuning. Smaller datasets constrained the ability of machine learning models to generalize effectively. Nevertheless, by using stable parameter settings, we achieved consistent and satisfactory performance across most datasets.

While a HITL approach remains indispensable for analyzing complex qualitative data, developing systems that support and enhance this collaboration is a promising direction for the future evolution of the CM methodology. Further application of the approach across diverse contexts will be necessary to fully understand its capabilities and identify opportunities for improvement.

In summary, our findings suggest that IdeaDistiller can significantly streamline a previously time-consuming step of the CM process, improve scalability, and maintain reasonable accuracy. Nonetheless, human validation remains essential to ensure the quality and relevance of results in real-world applications.

In this study, we created IdeaDistiller, a tool trained on previous CM studies. An important next step will be to use and test the tool in further real-life studies across different domains to enable continued evaluation with respect to both usability and the experience of time-saving.

Additionally, advances in explainable AI could be leveraged to improve the interpretability of clustering results by providing insights into how and why specific items are grouped together. For instance, techniques such as attention heatmaps, feature importance scores, or natural language explanations could help domain experts understand the rationale behind each cluster, thereby enhancing transparency, supporting critical evaluation, and increasing trust in the system’s outputs.

Overall, the continued development of semiautomated tools, coupled with human expertise, presents a compelling path forward for improving the efficiency, scalability, and methodological rigor of CM and other qualitative research methodologies.