The data used in this study were obtained from 3 well-established cohort studies, namely CHS, MESA, and WHI. Ethics approval (IRB17-2059) was granted by the Institutional Review Board of the Harvard Faculty of Medicine. Institutional Review Board approval was secured for access to all studies’ retrospective data. De-identified data was accessed through a secure cloud storage platform. Participants were not compensated for the use of their data in this study.

The CHS was a population-based longitudinal study initiated to determine the risk factors for the development and progression of clinically validated cardiovascular disease in adults aged 65 years and older. Beginning in 1989, the study enrolled 5888 participants from 4 US communities: Forsyth County, NC; Sacramento County, CA; Washington County, MD; and Pittsburgh, PA. The cohort consisted of two recruitment waves: the original cohort (1989-1990) and the African American cohort (1992-1993). Comprehensive baseline examinations were conducted, including medical history, physical examinations, laboratory tests, and others, with annual follow-ups to ascertain cardiovascular events [26].

The WHI was a long-term national health study that focused on strategies for preventing heart disease, breast and colorectal cancer, and osteoporotic fractures in postmenopausal women. Launched in 1991, the WHI involved multiple clinical trials and an observational study, enrolling a total of 161,808 women aged 50-79 years across 40 clinical centers throughout the United States. The participants were ethnically diverse, reflecting the demographic composition of the US population. Extensive data on lifestyle, health, and medical history were collected at baseline and at regular intervals throughout the study, creating a rich source of information for a variety of research endeavors [27].

The MESA study was a prospective cohort designed to delve into the prevalence and progression of subclinical cardiovascular disease among community-dwelling adults. MESA assessed a diverse, population-based sample of 6814 asymptomatic men and women aged between 45 and 84 years from 2000 to 2018. The participants were recruited from 6 field centers across the United States, including Wake Forest University, Columbia University, Johns Hopkins University, University of Minnesota, Northwestern University, and University of California – Los Angeles. The MESA cohort was made up of 38% White, 28% African-American, 22% Hispanic, and 12% Asian (primarily Chinese) individuals. Since its inception in July 2000, the study conducted 6 examinations, each occurring every 18 to 24 months [25].

The process of data extraction necessitated the gathering of documentation for variables within the CHS, MESA, and WHI studies. This information was procured from the Database of Genotypes and Phenotypes (dbGaP) [28]. We used dbGaP metadata to procure the following salient information for each variable within the study: (1) variable accession, (2) variable name, (3) variable description, and (4) dataset accession. While the variable name and variable description were not necessarily unique within or between studies, the variable accession was a unique identifier across all studies. We used the variable description strings as the semantic data in our model. From the raw variable description strings, we further extracted and removed the temporal period during which the variables were measured by parsing for key temporal terms, such as visit and exam. This data extraction process facilitated a comprehensive understanding of the variables’ conceptual characteristics, thereby providing the foundation for the subsequent data harmonization efforts.

We used the dataset accession and variable accession identifiers to access (1) variable metadata and (2) the patient-level data for the set of variables already extracted from dbGaP. To allow for relevant distribution comparisons between variables, we kept only continuous data by filtering variables using the continuous flag in the metadata.

Our study scope was primarily focused on the harmonization of continuous variables at the conceptual level. A “concept” in this context was defined as the underlying notion or theme that a variable represents, independent of the specific unit or time point of measurement. For example, a biomarker such as C-reactive protein, despite being reported in different units across different visits, was treated as having the same concept. Moreover, concepts were sometimes encoded in natural language or questionnaire form, rather than standard medical terms. We focused on conceptual level harmonization for several reasons. Researchers conducting multicohort studies are often interested in identifying all variables corresponding to a concept. Depending on the application, they might be interested in further refining this concept-level harmonization or also harmonizing the variable values across different units or collection time periods. Removing temporal information and units allowed us to focus on the essential meaning or theme underlying the variable, thereby facilitating the primary task of concept-level harmonization, which is manually challenging and resource intensive, paving the way for further data harmonization. Moreover, doing so enhanced comparability across studies, as variables with the same concept were treated as equivalent, irrespective of the units used. During the initial phase of data preprocessing, we streamlined variable descriptions by eliminating temporal information phrases. This practice not only simplified the descriptions but also augmented their comparability.

We also applied filters to variables according to their values. First, we removed variables with incomplete patient data. To preserve a significant portion of variables, we considered incomplete patient data at the subgroup level rather than the individual patient level. We defined patient subgroups using the anchor variables of age, race, and sex. Per the characteristics of the study populations and data availability, we defined 4 age buckets (≤59 years, 60 to 69 years, 70 to 79 years, and ≥80 years). For the categorical anchor variables, we used 2 predefined race categories (White and Black) and 2 predefined sex categories (female and male), yielding a total of 16 possible patient subgroups (calculated as 4×2×2). For each study, we removed variables that had no patient data for one or more subgroups, considering only patient subgroups that were represented in the cohort.

Second, we purged variables that had uniformly zero values across all patients. This removal was necessary as variables without variability do not offer predictive power and thus, contribute little to subsequent analyses. Subsequently, we identified variables within the same study with identical descriptions and treated them as a single entity. Rather than maintaining these as separate variables, we amalgamated their distribution vectors by computing their element-wise mean. This consolidation concurrently reduced redundancy and bolstered the statistical power and robustness of downstream analyses. The underlying principle driving these measures was the emphasis on the core conceptual content encapsulated within variables, a focus that lays the groundwork for a more efficient and meaningful process of data harmonization.

To assess the harmonization accuracy of SONAR, we manually created a set of gold standard labels. The process began with the curation of a concept list, consisting of common diseases, laboratory results, and medications, consistent with the goal of harmonization at the conceptual level (details in Multimedia Appendix 1, Section 4). With the concept list in hand, each of the 3 independent reviewers assigned raw variables from all 3 studies to the corresponding concepts based on their descriptions. Not all variables had labels, since the curated concept list was not comprehensive of all underlying concepts present in the studies. To ensure consistency and accuracy, we adopted a consensus-based approach for handling any discrepancies among the reviewers. In cases of disagreements, the reviewers discussed their rationales for their assignments, and through a process of discussion, literature review, and majority vote, they reached a consensus on the appropriate concept assignment. This rigorously prepared set of annotations, backed by consensus, formed our gold standard labels. In particular, each pair of variables corresponding to the same underlying concept formed a gold standard pair (ie, a pair of variables that a harmonization algorithm should map to each other). Such pairs consisted of variables from two different datasets (intercohort) or the same dataset (intracohort), since multiple variables from a single dataset could correspond to the same underlying concept. These labels offered a reliable standard against which we could validate our semantic learning and patient data learning techniques.

The proposed SONAR approach had 4 steps, including semantic learning, distribution learning, concatenation of the two learnings, and supervised training. Underpinning both semantic learning and distribution learning was the idea that variables with similar textual descriptions and patient-level value distributions were more likely to encode the same underlying concept.

We assessed the performance of the proposed SONAR approach both within individual cohorts (intracohort) and between different cohorts (intercohort). We considered the area under the curve (AUC) of the receiver operating characteristic curve as the overall metric for harmonization accuracy. Specifically, for each underlying concept, we first computed the cosine similarity of the embedding vectors for known concept pairs (true positives) and an equal number of randomly selected concept pairs (false positives), where the cosine similarity measured the cosine of the angle between 2 vectors in a multidimensional space, effectively quantifying how similar they are. The AUC was then calculated, summarizing the overall accuracy of our method across varying decision thresholds for a given underlying concept. The overall AUC was the average of the concept-level AUC values. To reduce the effect of outliers within the sampled negative pairs, we averaged the 3 overall AUC values calculated for 3 sets of sampled negative pairs.

We also evaluated the performance of SONAR on hard concepts, defined to be concepts for which the benchmark SapBERT AUC was below the threshold of 0.900. The hard AUC was the average of the concept-level AUC values for the hard concepts. This was an important metric for demonstrating the added value of distribution learning and more broadly the complementary effects of the two forms of learning.

To further scrutinize the performance of our method, we reported the top-k accuracy (acc@k) for mapping of codes from Cohort A to Cohort B, where the cohorts were identical for intradataset mapping. For variable a in Cohort A, we let Ba be the set of variables within Cohort B with embeddings that had the largest cosine similarity with variable a, and we let Ga be the set of variables within Cohort B that were in a positive gold standard pair with variable a. Then, the acc@k for an underlying concept for the mapping from Cohort A to Cohort B was the number of codes a such that Ba∩Ga≠0 divided by the total number of gold standard variables in Cohort A corresponding to the underlying concept. The acc@k for the mapping from Cohort A to Cohort B was then the average of the acc@k across all underlying concepts. This prevented dominance by underlying concepts with many corresponding gold standard variables. To calculate the intracohort acc@k, we first computed the intracohort acc@k for each underlying concept by averaging the acc@k for that underlying concept across all 3 intracohort comparisons (intra-CHS, intra-MESA, and intra-WHI). Underlying concepts were only averaged over the comparisons for which they were relevant, so the intracohort acc@k for a concept that had gold standard pairs in only CHS and MESA was be the average of the CHS acc@k and the MESA acc@k for the concept in question. Then, the intracohort acc@k was the average of the acc@k across all underlying concepts. The intercohort acc@k was calculated similarly, except across the 6 intercohort mappings (CHS to MESA, CHS to WHI, MESA to CHS, MESA to WHI, WHI to CHS, WHI to MESA). Finally, the overall acc@k was calculated across the 9 total mappings for each underlying variable. We obtained acc@k values for k values of 1, 3, 5, 10, 20 in order to provide additional insight into SONAR’s effectiveness at capturing true positives at different thresholds. These rigorous evaluations allowed us to confidently assert the efficacy of our harmonization method in both intracohort and intercohort settings.

For the supervised portion of our method, we used 2-fold cross-validation. We pooled cosine similarity values from both rounds of training before AUC calculations and averaged the 2 resulting cosine similarity values for non–gold standard pairs before acc@k calculations. To provide a comprehensive understanding of our method’s performance, we compared the AUC and other accuracy metrics obtained by SONAR with those obtained when using semantic learning (BioBERT [Bidirectional Encoder Representations from Transformers for Biomedical Text Mining], CODER, SapBERT) or distribution learning alone. We also obtained metrics for the concatenated semantic portion of SONAR (ie, CODER concatenated with SapBERT) to further highlight the added value of distribution learning. This comparative approach allowed us to illuminate the relative strengths and contributions of the individual components of our method and the added value achieved by their combination.

The proposed SONAR approach had 4 steps, including semantic learning, distribution learning, concatenation of the two learnings, and supervised training (Figure 1). We extracted metadata and semantic data from the dbGaP for 14,717 CHS variables, 22,147 MESA variables, and 6207 WHI variables. Filtering for continuous, complete, and nonzero variables using metadata and patient-level data from Service WorkBench, as well as consolidating variables with identical semantic descriptions yielded 2076 CHS variables, 2525 MESA variables, and 1328 WHI variables. Patient data was available for 12 subgroups, 16 subgroups, and 6 subgroups for CHS, MESA, and WHI, respectively. This yielded distribution vectors of length 36, 48, and 18 for intra-CHS, intra-MESA, and intra-WHI harmonization, respectively. Based on overlapping patient subgroups between the cohorts, we used distribution vectors of length 36, 12, and 18 for intercohort CHS-MESA, CHS-WHI, and MESA-WHI harmonization, respectively.

We identified a total of 123 concepts with continuous data, consisting of 112 laboratory test concepts, 6 disease concepts, and 5 medication concepts. A total of 531 variables across all cohorts were identified as gold standard representations of these concepts. These yielded 606, 318, and 89 gold standard concept pairs for intradataset harmonization evaluation within the CHS, MESA, and WHI, respectively. For interdataset harmonization evaluation, we had 352, 325, and 133 gold standard concept pairs for CHS-MESA, CHS-WHI, and MESA-WHI, respectively. Detailed numerical summaries of gold standard labels are provided in Table 1.

Supervised SONAR achieved a strong performance across all intracohort AUC (Figure 2 and Table S1 in Multimedia Appendix 1) and acc@k (Figure 3 and Table S2 in Multimedia Appendix 1) measures, exceeding or meeting all benchmark comparisons. The number of hard concepts for each intracomparison was 13 concepts, 5 concepts, and 5 concepts for intra-CHS, intra-MESA, and intra-WHI, respectively. It is notable that the distribution-only AUC was significantly higher than the semantic-only methods for the intra-CHS and intra-WHI hard concepts, illustrating the advantage of incorporating both semantic and distribution learning in SONAR. While the addition of supervised training only improved overall AUC performance for the intra-WHI comparison, it improved intracohort AUC performance on hard concepts for all 3 intracohort comparisons, exceeding all benchmark methods. The addition of supervised training also improved acc@k performance of SONAR across different values of k. Across intracohort evaluations, distribution learning provided a clear added value to semantic learning, in spite of a weaker distribution-only performance in comparison to the semantic components of SONAR (CODER only, SapBERT only, CODER + SapBERT). Moreover, there was not a single best semantic learning method between CODER and SapBERT using the various AUC and acc@k metrics, providing support for the use of both semantic learning methods in SONAR.

Supervised SONAR also achieved a consistently high performance in intercohort harmonization evaluation, exceeding or meeting all benchmark comparisons except for acc@3 and acc@20 (Figure 3). The number of hard concepts for each intracomparison was 4 concepts, 11 concepts, and 5 concepts for the CHS-MESA, CHS-WHI, and MESA-WHI comparisons, respectively. Similar to intracohort harmonization, supervised training improved AUC performance on hard concepts for all 3 comparisons, exceeding all benchmark methods. In contrast with intracohort harmonization, supervised training also improved AUC performance on all concepts to above 0.99 for all 3 intercohort comparisons (Figure 4). Notably, the CODER only and SapBERT only AUC values were higher for intercohort harmonization as compared to intracohort harmonization because identical variable descriptions were allowed for intercohort semantic learning.