This is an openaccess article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.
Health care organizations are collecting increasing volumes of clinical text data. Topic models are a class of unsupervised machine learning algorithms for discovering latent thematic patterns in these large unstructured document collections.
We aimed to comparatively evaluate several methods for estimating temporal topic models using clinical notes obtained from primary care electronic medical records from Ontario, Canada.
We used a retrospective closed cohort design. The study spanned from January 01, 2011, through December 31, 2015, discretized into 20 quarterly periods. Patients were included in the study if they generated at least 1 primary care clinical note in each of the 20 quarterly periods. These patients represented a unique cohort of individuals engaging in highfrequency use of the primary care system. The following temporal topic modeling algorithms were fitted to the clinical note corpus: nonnegative matrix factorization, latent Dirichlet allocation, the structural topic model, and the BERTopic model.
Temporal topic models consistently identified latent topical patterns in the clinical note corpus. The learned topical bases identified meaningful activities conducted by the primary health care system. Latent topics displaying nearconstant temporal dynamics were consistently estimated across models (eg, pain, hypertension, diabetes, sleep, mood, anxiety, and depression). Several topics displayed predictable seasonal patterns over the study period (eg, respiratory disease and influenza immunization programs).
Nonnegative matrix factorization, latent Dirichlet allocation, structural topic model, and BERTopic are based on different underlying statistical frameworks (eg, linear algebra and optimization, Bayesian graphical models, and neural embeddings), require tuning unique hyperparameters (optimizers, priors, etc), and have distinct computational requirements (data structures, computational hardware, etc). Despite the heterogeneity in statistical methodology, the learned latent topical summarizations and their temporal evolution over the study period were consistently estimated. Temporal topic models represent an interesting class of models for characterizing and monitoring the primary health care system.
Electronic medical record (EMR) systems are increasingly being adopted in clinical settings across the globe [
Several methods can be used to estimate a topic model, given a document collection, and to characterize the evolution of latent topical bases over time. Latent Dirichlet allocation (LDA) [
The objective of this study was to compare the performance of several temporal topic modeling methodologies fitted to a corpus of primary care clinical notes. We compared the following temporal topic modeling methodologies: NMF, LDA, STM, and BERTopic. We examined (1) the overall matrix of pertopic word probabilities estimated over the corpus and (2) the multivariate time series structures describing the evolution of latent topical prevalence weights (k=1...K) over discrete times (t=1...T). We compared the methods using a data set of longitudinal primary care clinical notes collected over 5 years (20112015) in Ontario, Canada.
Topic models use statistical information regarding documentword cooccurrence frequencies to learn meaningful latent variable representations from a corpus. Each document in the collection (d=1...D) is represented as a highdimensional lengthV vector (v=1...V), where each element is a count of the number of times a particular word or token (v) in an empirical vocabulary is observed in a particular document (d). We represented the collection of documentspecific termfrequency vectors into a matrix X of dimension D*V, called the DTM. The DTM is a large, sparse matrix. However, the matrix is overdetermined because many of the rows (representing documentspecific termfrequency vectors) and columns (representing word or token occurrence frequency over all documents in the corpus) demonstrate strong intercorrelations. Dimensionreduction techniques, such as topic models, use intercorrelated statistical semantic information to estimate meaningful thematic representations from document collections. Topic models learn (1) clusters of intercorrelated words describing the topical content of the corpus and (2) clusters of correlated documents sharing latent topical concepts.
The most challenging and subjective aspect associated with construction of the DTM involves specification of the vocabulary or dictionary (v=1...V) encoding the column space of the matrix. A priori constructed lexicons or dictionaries (of dimension V) can be used to determine the study vocabulary. Specification of appropriate domainspecific dictionaries would be tasked with subject matter experts on the research team. Alternatively, an entirely computational approach could specify a text tokenization or normalization pipeline and computationally parse the input character sequences into a finite number of tokens.
In this study, we adopted a hybrid approach to vocabulary or dictionary specification. We began by tokenizing the clinical notes on whitespace boundaries (spaces, tabs, newlines, carriage returns, etc). We normalized tokens using lowercase conversion and removed all nonalphabetic characters. We removed tokens with a character length ≤1. Finally, we sorted the list of tokens or words by decreasing occurrence frequency and manually reviewed the sorted list of tokens. Our manual review identified V=2930 distinct tokens for inclusion in our final vocabulary. The total number of tokens in the corpus was 3,003,583. The tokens chosen for inclusion in our final dictionary or vocabulary were mainly medical terms with precise semantic meanings (disease names, disease symptoms, drug names, medical procedures, medical specialties, anatomical locations, etc). We excluded stop words or tokens (ie, syntactic or functional tokens with little clinical semantic meaning). Words with low occurrence frequency were excluded for computational considerations. All text processing was conducted using R (R Foundation for Statistical Computing; version 3.6).
NMF estimates latent topical matrices using the documentword cooccurrence statistics contained in the empirical DTM. NMF factorizes the D*V dimensional DTM into 2 latent submatrices of dimensions D*K (θ) and K*V (Φ). The DTM (X) consists of nonnegative integers (ie, word frequency counts), whereas the learned matrices (θ,Φ) consist of nonnegative real values. Mathematically, the NMF objective involves learning optimal values of the latent matrices (θ,Φ) that best approximate the input data set (X ≈ θΦ), subject to the constraint that the learned matrices contain nonnegative values.
We selected a least square loss function to train the NMF model. The objective function specifies that the observed data elements are approximated in a Kdimensional bilinear form
Post hoc, the row vectors constituting both θ and Φ, can be normalized by dividing by their respective row sums. The resulting normalized vectors can be interpreted as compositional or probability vectors (ie, each normalized row of θ and Φ contains nonnegative entries that sum to 1, rowwise). The row vectors of the matrix Φ encode a set of k=1...K pertopic word probabilities or proportions (estimated over a discrete set of v=1...V words in the empirical corpus vocabulary). The row vectors of the matrix θ encode a set of d=1...D perdocument topic proportions (estimated over a discrete set of k=1...K latent dimensions), encoding the affinity a given document has for a particular topic.
For each document d=1...D, assume we observe a time stamp that allows us to associate each document (and latent embedding) with a Tdimensional indicator variable denoting the observation time (t=1...T). We estimated a Kdimensional multivariate mean topical prevalence vector for each design point, t=1...T. This resulted in a multivariate time series structure (a T*K dimensional matrix). Each column (k=1...K) of the matrix is a length T time series that described the evolution of a latent topical vector.
The sklearn.decomposition.NMF() function in the Python SKLearn package (version 0.24.2) was used to fit the NMF topic model.
LDA is a probabilistic topic model. Probabilistic topic models assume that a document comprises a mixture of topics. These (latent) topics represent a probability distribution over a finite vocabulary of words or tokens. Topic models can also be described as admixture models. Each document is a soft mixture of topics (k=1...K), where a topic is itself a probability distribution over words in the vocabulary (v=1...V). A graphical model describing LDA is shown in
The LDA graphical model also describes a generative process for creating a single document in the corpus. This can be succinctly described using the following sampling notation [
To generate a document, we begin by sampling the pertopic word distributions from a Dirichlet distribution parameterized by a V dimensional prior concentration parameter (β). Topical vectors (k=1...K) are shared over the collection of documents.
Next, for each document d=1...D in the collection, we sample the perdocument topic distribution from a Dirichlet distribution parameterized according to a Kdimensional prior concentration parameter (α).
For each word in each document, we sample a topical indicator variable, z_{d,n}. This variable takes an integer value between 1 and K and signifies the pertopic word distribution from which a specific word, w_{d,n}, is chosen. The index n denotes the n^{th} word in a variable length document (n=1...N_{d}).
Finally, we draw a single word token, w_{d,n}, from the topical distribution associated with z_{d,n}. The word indicator is an element v=1...V in our empirical dictionary or vocabulary.
The statistical inference problem associated with probabilistic topic modeling involves inverting the sampling process and learning modeldefined latent parameters given the observed text data. The latent variables indicate which words are assigned to which topical indicators (z), which documents have an affinity for which topics (θ), and which words cooccur with high likelihood under which topics (Φ). The latent parameters associated with an LDA topic model are typically estimated using Bayesian statistical machinery (Gibbs sampling [
A multivariate transformation of the matrix of perdocument topical prevalence weights generates a multivariate time series data structure. This object is of dimension T*K, where each column k=1…K represented a univariate topical time series of length T. This series describes the evolution of latent topical vectors over our study period.
The sklearn.decomposition.LatentDirichletAllocation() function in Python SKLearn (version 0.24.2) was used to fit the LDA topic model.
Graphical model representation of the latent Dirichlet allocation topic model.
The STM is another type of probabilistic topic model. The STM extends the LDA topic model, allowing latent matrices of (1) perdocument topical prevalence weights or (2) pertopic word proportions to vary according to a generalized linear model parameterization [
To generate a document under STM, we begin by sampling the pertopic word distributions from an (interceptonly) multinomial logit model (where multinomial logit regression parameters are given sparse “gammalasso” prior) [
Next, we sample the perdocument topic distribution from a logisticnormal distribution parameterized in terms of a mean vector and covariance matrix. γ represents a D*T dimensional design matrix encoding the time point (t=1...T) under which the document (d=1...D) was observed. The vector γ is a matrix of dimension T*K and encodes discrete time effects on each of the perdocument topical prevalence weights (a length K vector for each document d=1...D). Finally, Σ is a K*K dimensional covariance matrix that encodes correlations between topical prevalence vectors (parameterized under a logisticnormal model).
For each word (n=1...N_{d}) in each document (d=1...D), we sample a topical indicator variable z_{d,n}. This variable takes an integer value between 1 and K and signifies the pertopic word distribution from which a specific word, w_{d,n}, is chosen. It must be noted that the upper limit N_{d} suggests that the number of words used for any given document (d) can vary.
Finally, we draw a single word or token, w_{d,n}, from the topical distribution associated with z_{d,n}. The word indicator is an element v=1...V in our empirical dictionary or vocabulary.
The framework for STM naturally allows for the estimation of temporal effects on topical prevalence weights. In our study, discrete time effects on topical prevalence can be interpreted using the coefficient matrix (γ) from the fitted logisticnormal model. As the temporal effects are encoded in a Bayesian regression modeling framework, we can also compute inferential measures (posterior means, highest posterior density intervals, etc). The singlestage inferential mechanism encoded in STM is a clear strength over earlier NMF and LDA models.
We used the stm() function in the STM package in R to fit the STM to our study data.
Graphical model representation of the structural topic model.
Recently, researchers have developed topic models that integrate neural architectures and related techniques for model specification and learning. These neural topic models represent a different class of topic models compared with those introduced previously. Examples of recently developed neural topic models include top2vec [
BERTopic begins with embedding documents empirically observed in the study corpus into a latent embedding space. Many methods exist for embedding discrete linguistic units (words, sentences, paragraphs, documents, etc) into an embedding space. For example, words can be embedded in a vector space using word2vec [
Each document (d=1...D) is embedded in a vector space, typically of a few hundred dimensions. The uniform manifold approximation and projection (UMAP) algorithm [
Clusters (k=1...K) of semantically related documents were identified. Scores over words v=1...V in the vocabulary were computed using clusterspecific TFIDF weights. If a cluster consisted of semantically focused documents, and hence words, we expect to observe coherent and meaningful words identified via TFIDF scoring. The proportion of documents assigned to each cluster during a specific period (t=1...T) can be used to generate a T*K dimensional multivariate time series structure, depicting the evolution of latent topic over our study period.
We fitted the BERTopic model using default hyperparameter settings. The BERTopic pipeline requires (1) specification of a document embedding algorithm (in our case, the MPNet sentence transformer model [
We used the Python package bertopic to fit BERTopic models.
We used simple counts and percentages to describe the characteristics of our study sample. We described the number of unique patients and number of unique clinical notes. Each patient in our sample was a “highuser” of the primary care system, in the sense they generated at least one encounter/note for each of the twenty quarterly time periods between 20112015. We described the distribution of the number of notes per patients. We described demographic characteristics of the sample (age/sex distributions).
When fitting the NMF, LDA, and STM models, we constructed a DTM whose row dimension corresponded to the number of unique patients in the sample (ie, 1727 unique patients) multiplied by the number of distinct time periods (
For each of the NMF, LDA, STM, and BERTopic models, we constructed a K*T dimensional multivariate time series matrix (this is the transpose of the T*K data structure described earlier). Each row corresponds to a latent topic vector and each column corresponds to a specific quarterly time period. A row vector is a length T time series describing the evolution of a latent topical vector across the study periods. Each column corresponds to a distribution over topics at a particular period (ie, described which topics are most important at a given period). For each row k=1...K, we report the top 5 words loading most strongly on a given topic. The cluster of words was semantically correlated and described the essence of the latent topical vector. A heatmap was used to visualize this highdimensional multivariate time series structure; and we hierarchically clustered the rows of the matrix using a Euclidean distance metric and Ward agglomeration method (a dendrogram was used to visualize the cluster structure of the topical series).
The topical structure of each of the NMF, LDA, STM, and BERTopic model fits was described in terms of the top 5 words loading most strongly on each of the k=1...K latent topics. In other words, the topical structure of each model can be described in terms of a “bag” of 250 words or tokens. We investigated the topical diversity of the model fits. Topical diversity was calculated in terms of the number of unique words in the bag of 250 total words. Furthermore, we investigated the top 5 most frequently occurring words in the “bag” describing each model fit. The redundantly occurring words in the topical summaries provided a rough approximation of the semantic concepts that the models repeatedly identified as important.
We investigated several measures of topical coherence for the NMF, LDA, STM, and BERTopic models. We considered the “UMASS,” “UCI,” and normalized pointwise mutual information (“NPMI”) metrics described in the surveys of Roder et al [
We used a setbased measure of concordance, the Jaccard coefficient, to assess similarities or differences in the topical structure describing the NMF, LDA, STM, and BERTopic models. Each model was described in terms of a “bag” of 250 words or tokens (ie, k=50 topics, described in terms of their top 5 most probable words); consider 2 models generating bags of words or tokens, b_{0} and b_{1}. The Jaccard coefficient is defined as the cardinality of the intersection of b_{0} and b_{1} divided by the cardinality of the union of b_{0} and b_{1}. In mathematical notation, the Jaccard coefficient is expressed as follows:
Finally, we described the wall time (in seconds or minutes) required to fit each of the NMF, LDA, STM, and BERTopic models. We also discussed the computational issues associated with hyperparameter tuning of each of the models.
This study used a retrospective closed cohort design. Clinical notes were obtained from primary care EMR systems geographically distributed across Ontario, Canada. We included all clinical notes written by the patient’s primary care provider between January 01, 2011, and December 31, 2015. We discretized time into quarterly strata (JanuaryMarch; AprilJune; JulySeptember; and OctoberDecember). Patients were excluded if they did not have at least one clinical note in each of the 20 quarterly strata over the study period. Hence, the selected sample of patients reflects a unique set of individuals who frequently engaged with the primary health care system.
Our document collection contained 160,478 clinical notes from 1727 patients. The 1727 patients received primary care services from 1066 unique primary care physicians at 40 unique primary care clinics (geographically distributed across Ontario, Canada). The median age of the patients was 68 (IQR 5580) years and ranged from 20 to 103 years (age statistics were calculated using study baseline as a reference date, January 1, 2011). Female patients were observed more frequently than male patients (1157/1727, 67% vs 570/1727, 33%).
The initial notelevel DTM had dimensions of 160,478 rows (one row for each clinical note in the corpus) by 2930 columns (one column for each unique word or token in the corpus). The corpus comprised 3,003,583 tokens. The DTM was >99% sparse (ie, it contained almost all zero elements). We also constructed a patientquarter–level DTM by aggregating notes observed on the same patient within a quarter. This DTM had dimensions of 1727×20=34,540 rows by 2930 columns and was >98% sparse. The top 25 most frequently occurring words in the analytic corpus are listed in
Descriptive statistics for study sample, at notelevel and patientlevel unit of analysis.
Characteristic  Unique notes (n=160,478), n (%)  Unique patients (n=1727), n (%)  



2040  9713 (6.1)  107 (6.1) 

4065  63,588 (39.6)  675 (39.1) 

6585  63,839 (39.8)  704 (40.8) 

>85  23,338 (14.5)  241 (14) 



Male  51,530 (32.1)  570 (33) 

Female  108,948 (67.9)  1157 (67) 



2011  28,012 (17.5)  —^{a} 

2012  31,220 (19.5)  — 

2013  33,676 (21)  — 

2014  33,756 (21)  — 

2015  33,814 (21)  — 
^{a}Not applicable.
Top 25 most frequently occurring tokens or words in the final analytic primary care clinical note corpora (N=3,003,583).
Token or word  Occurrence frequency, n (%) 
pain  88,132 (2.93) 
mg  65,612 (2.18) 
inr  52,970 (1.76) 
bp  50,751 (1.69) 
back  43,556 (1.45) 
dose  29,861 (0.99) 
feels  24,736 (0.82) 
rx  23,211 (0.77) 
chest  22,256 (0.74) 
meds  20,914 (0.7) 
referral  19,409 (0.65) 
work  19,398 (0.65) 
wt  19,322 (0.64) 
feeling  17,415 (0.58) 
blood  16,121 (0.54) 
symptoms  15,905 (0.53) 
prn  15,706 (0.52) 
urine  14,633 (0.49) 
bw  13,779 (0.46) 
lab  13,543 (0.45) 
clear  13,271 (0.44) 
knee  12,677 (0.42) 
pharmacy  12,503 (0.42) 
sleep  12,331 (0.41) 
prescription  11,945 (0.4) 
We comparatively evaluated inferences obtained from fitting the NMF, LDA, STM, and BERTopic models to our primary care clinical note corpus. For each model, we varied the number of topics (K={25,40,45,50,55,60,75}) and observed similar inferences at various levels of the model complexity parameter (K). When K was too small, distinct semantic topics tended to be grouped together, whereas when K was too large, semantically similar topics tended to be split into arbitrary clusters (resulting in an overclustering effect). Using human judgment evaluation, we determined that a model complexity of K=50 topics balanced a parsimonious, while simultaneously expressive, characterization of the clinical document corpus. For each of the NMF, LDA, STM, and BERTopic models, we reported the results assuming K=50 latent topics.
A summary of the distribution of words over the k=1...50 latent topics (for each of the 4 models under comparison) is given in
Each of the 4 latent temporal topic models learned a meaningful representation of the primary care clinical notes corpus. In the following paragraphs, we discuss (1) topics consistently estimated across models that demonstrated constant trends in topical prevalence across quarterly periods and (2) topics consistently estimated across quarterly periods that demonstrated interesting seasonal patterns.
Each of the fitted models consistently identified the following latent primary care topical constructs (and these topics show constant patterns across quarterly periods): sleep (NMF=Topic−45; LDA=Topic2 or Topic31; STM=Topic11; BERTopic=not applicable); mental health, for example, mood, anxiety, and depression, (NMF=Topic33; LDA=Topic22; STM=Topic19; BERTopic=Topic16); pain (NMF=Topic1; LDA=Topic39, Topic36, Topic14, Topic49, Topic34, or Topic37; STM=Topic8; BERTopic=Topic9 or Topic39); blood pressure control and monitoring (NMF=Topic36; LDA=Topic9; STM=Topic21; BERTopic=Topic31); respiratory disease, for example, cough, throat, chest, fever, etc (NMF=Topic46; LDA=Topic13; STM=Topic46; BERTopic=Topic1), smoking (NMF=Topic31; LDA=Topic32; STM=Topic44; BERTopic=Topic38); diabetes, for example, blood, sugar, insulin, fbs, etc (NMF=Topic5; LDA=Topic43; STM=Topic42; BERTopic=Topic8); pharmaceutical prescription management (NMF=Topic26; LDA=Topic40; STM=Topic9; BERTopic=Topic36 or Topic5); and annual influenza vaccination programs (NMF=Topic6; LDA=Topic29; STM=Topic36; BERTopic=Topic50). These thematic areas represented archetypical patients, conditions, or roles encountered in the primary health care system. The consistent extraction of latent themes (represented as semantically correlated word clusters) suggests that each model can leverage information regarding wordcontext cooccurrence to learn meaningful patterns from a large unstructured clinical document corpus.
For certain learned topics, seasonal harmonic patterns were stably estimated over the study period. For example, the annual influenza vaccination program consistently occurred in the fall or winter months of the study (NMF=Topic6; LDA=Topic29; STM=Topic36; BERTopic=Topic50). Similarly, annual spikes in respiratory diseases (cough, cold, influenza, etc) are identified as achieving peaks in the winter months and lows in the summer months (NMF=Topic46; LDA=Topic13; STM=Topic46; BERTopic=Topic1). These findings are illustrated in
A heat map of the multivariate time series structure associated with the nonnegative matrix factorization temporal topic model.
A heat map of the multivariate time series structure associated with the latent Dirichlet allocation temporal topic model.
A heat map of the multivariate time series structure associated with the structural topic model temporal topic model.
A heat map of the multivariate time series structure associated with the BERTopic temporal topic model.
Dendrograms displaying the clustering structure of the latent multivariate time series objects learned from nonnegative matrix factorization model (A), latent Dirichlet allocation model (B), structural topic model (C) and BERTopic model (D).
Descriptive time series plots characterizing the seasonal evolution of annual influenza program topic, as estimated by nonnegative matrix factorization model (A), latent Dirichlet allocation model (B), structural topic model (C) and BERTopicmodels (D).
Descriptive time series plots characterizing the seasonal evolution of the respiratory disease topic, as estimated by nonnegative matrix factorization model (A), latent Dirichlet allocation model (B), structural topic model (C) and BERTopicmodels (D).
When investigating the topranked words associated with perword topic distributions in
We explored the semantic coherence of NMF, LDA, STM, and BERTopic models using the following metrics: “UMASS,” “UCI,” and “NPMI” (
To investigate the differences and similarities in the fitted topic model, we used the Jaccard coefficient (
The time required to train each model was reported. For NMF, LDA, and STM models, we used a single central processing unit (although Python SKLearn implementations of decomposition models can be parallelized). For the BERTopic model, we used a single graphics processing unit for embedding documents and a single central processing unit for dimensionality reduction (UMAP) and clustering (HDBSCAN). Under these settings, the time required to fit the NMF, LDA, STM, and BERTopic models was 237 seconds, 67 seconds, 879 seconds (14.7 minutes), and 2624 seconds (43.7 minutes), respectively. The computational requirements of the BERTopic model exceeded those of the other models, particularly the highly optimized NMF or LDA implementations in Python SKLearn.
The most frequently occurring tokens observed in each of the bags of 250 words describing the topical structure of latent Dirichlet allocation (LDA), nonnegative matrix factorization (NMF), structural topic model (STM) and BERTopic model fits (and their occurrence counts in the bag).
Word or token  Topic model  

NMF (n)  LDA (n)  STM (n)  BERTopic (n) 
Word or token1  head (4)  back (9)  back (5)  inr (11) 
Word or token2  mg (4)  bp (6)  mg (5)  mg (9) 
Word or token3  ccac (3)  pain (6)  pain (5)  lab (5) 
Word or token4  diabetes (3)  chest (3)  bp (4)  prescription (5) 
Word or token5  feeling (3)  feels (3)  feels (3)  dose (4) 
Topical coherence measures (“UMASS,” “UCI,” and normalized pointwise mutual information [“NPMI”]) estimated on each of the nonnegative matrix factorization (NMF), latent Dirichlet allocation (LDA), structural topic model (STM) and BERTopic models.
Topical coherence measure  Topic model  

NMF  LDA  STM  BERTopic 
UMASS  −2.522  −2.488  −2.372  −2.591 
UCI  1.220  0.987  1.192  1.405 
NPMI  0.183  0.149  0.190  0.230 
Jaccard coefficient metrics of setbased concordance between fitted topic models: nonnegative matrix factorization (NMF), latent Dirichlet allocation (LDA), structural topic model (STM), and BERTopic.

NMF  LDA  STM  BERTopic 
NMF  —^{a}  —  —  — 
LDA  0.526  —  —  — 
STM  0.491  0.577  —  — 
BERTopic  0.343  0.286  0.329  — 
^{a}Not applicable.
In this study, we compared several distinct methodologies (ie, NMF, LDA, STM, and BERTopic) to estimate temporal topic models from a large collection of primary care clinical notes. Despite differences in the underlying statistical methodology, models often converged on a consistent latent characterization of the corpus. Furthermore, the temporal evolution of latent topics was reliably extracted from each of the NMF, LDA, STM, and BERTopic models.
Clinically, our data set represented highusers of the primary care system. Many of the latent topics emerging from this analysis are consistent with a highuser archetype, for example, family counseling or social work, mood disorders, anxiety or depression, chronic pain, arthritis and musculoskeletal disorders, neurological conditions, cardiovascular disease and hypertension, diabetes, cancer screening (breast, cervical, colorectal, and prostate), laboratory requisitions and blood work, diagnostic imaging, and pharmaceutical or prescription management. Topic models also identified numerous acute health conditions as important latent themes, such as cough, cold and other respiratory infections, urinary tract infections, skin conditions, and wound care. NMF, LDA, STM, and BERTopic models each consistently captured (1) annual primary care influenza programs and (2) seasonal respiratory conditions, demonstrating predictable seasonal variation. Findings regarding primary care use patterns, extracted solely from clinical text data, were largely corroborated by provincial reporting based on structured administrative data [
We observed that disparate statistical methodologies for estimating temporal topic models generated a concordant or consistent latent representation. We interpreted this to mean that as the signaltonoise ratio increases in a given clinical text data set, the subtle choice of statistical methodology seems to matter less, and any of these methods would extract a meaningful latent representation of the primary care corpus. For smaller corpora, where worddocument cooccurrence statistics are less certain, this hypothesis may not hold.
Furthermore, subtle or nuanced differences in model representations emerged, which may lead analysts to favor specific modeling strategies in particular settings. For example, consider
Because of the different statistical principles associated with each temporal topic modeling methodology, each method is associated with its own strengths and weaknesses. We have elaborated on the methodological and computational issues associated with each class of models.
First, NMF is the most mature and seemingly parsimonious methodology for topic modeling. NMF is strongly rooted in linear algebraic principles and is fundamentally based on the constrained optimization of a simple least squares objective function. Vanilla NMF is a wellstudied statistical methodology and many efficient computational routines exist for estimating NMF models. NMF is flexible and can be readily extended. Possible model extensions can be viewed as discrete tunable hyperparameters in the model fitting process. Berry et al [
LDA and STM are Bayesian topic models. LDA was developed as a fully Bayesian extension of existing linear algebraicbased (eg, latent semantic analysis) and maximum likelihoodbased (eg, probabilistic latent semantic indexing) techniques for topic modeling [
BERTopic represents the most novel approach to topic modeling [
We attempted to be transparent with respect to how our final vocabulary of words or tokens was selected and accordingly the DTMs were constructed for this study. Different computational pipelines could have been used to preprocess our clinical text corpus. For instance, we could have used different strategies for tokenization, lemmatization, stemming, stopword removal, and frequencybased word or token removal. Different text preprocessing pipelines would ultimately lead to different DTM structures (with different vocabularies). Further research is needed to better understand the implications of these text preprocessing decisions on downstream study inferences.
Each topic model considered in this study requires specification of hyperparameters that govern the aspects of model fitting. Fitting these topic models is computationally intensive for large input data sets. We focused mainly on the stability and robustness of inferences with respect to model complexity (K), a common hyperparameter across all models. We did not explore the stability of the inferences across other modelspecific hyperparameters.
We did not consider all possible methods for estimating temporal topic models in this study. Bespoke NMF and LDA variants exist that are applicable for estimating temporal topic models. Sequential NMF [
These works have led us to consider several possible ways of extending different topic modeling frameworks, including Bayesian NMF with documentlevel covariates (similar to the STM extension of LDA), neural matrix factorization with (nontemporal) covariates, LDA or STM extensions that allow perdocument topical prevalence weights to vary according to a flexible generalized linear mixed model or multilevel model (for modeling dependencies introduced because of the complex design or sampling mechanism by which documents are created), and computational methods for improving statistical inference (eg, interval estimation and hypothesis testing) when engaging with temporal topic models (eg, resampling methods, bootstrap, and multiple outputation).
In this study, we compared several statistical techniques for estimating temporal topic models from primary care clinical text data. Different temporal topic models have unique strengths and weaknesses owing to their underlying statistical properties. Nonetheless, each model consistently estimated a latent variable representation of a primary care document collection, which meaningfully characterized highuse primary care patients and their longitudinal interactions with the primary health care system. As the adoption of EMRs increases and health care organizations amass increasingly large volumes of clinical text data, temporal topic models may offer a mechanism for leveraging unstructured clinical text data for characterization and monitoring of primary care practices and systems.
document term matrix
electronic medical record
hierarchical densitybased spatial clustering algorithm of applications with noise
latent Dirichlet allocation
nonnegative matrix factorization
normalized pointwise mutual information
structural topic model
termfrequency inversedocument frequency
uniform manifold approximation and projection
This study was supported by funding provided by a Foundation Grant (FDN 143303) from the Canadian Institutes of Health Research. The funding agency had no role in the study design; collection, analysis, or interpretation of data; writing of the report; or decision to submit the report for publication. Dr Austin is supported by a MidCareer Investigator Award from the Heart and Stroke Foundation.
None declared.