We developed 2 transformer-based language models—BERTDrug and CharBERTDrug—to detect misspelled drug names (Figure 1). We fine-tuned the models on an augmented dataset derived from RxNorm [25] and evaluated their performance on both a held-out RxNorm-derived test set (internal validation) and drug names from the LTCDC database [20] (external validation). A dictionary-based method (SpellChecker) and 2 machine learning–based methods (fastText_ML and BioWordVec_ML) served as the baselines.

The Institutional Review Board (IRB) of Boston University Chobanian & Avedisian School of Medicine (H-44363) reviewed the protocol for this study and determined the analysis of secondary data to be human subjects research exempt from IRB review. The IRB granted a waiver of HIPAA (Health Insurance Portability and Accountability Act) Authorization for Research under 45 CFR 164.512(i)(2)(ii). All methods were carried out in accordance with relevant guidelines and regulations.

We fine-tuned 2 transformer-based language models, CharacterBERT_medical and BERT_medical [28], to the task of detecting misspelled drug names.

Both CharacterBERT_medical and BERT_medical models are variants of the BERT (Bidirectional Encoder Representations from Transformers) model [29]. BERT is a state-of-the-art language model for natural language processing tasks such as question answering, text classification, and machine translation [30]. It builds on the Transformer architecture and leverages the self-attention mechanism for efficient contextual understanding [31]. BERT uses a subword vocabulary for tokenization and word/token embedding. It learns the embeddings bidirectionally, from both left to right and right to left of the input sequences of tokens. BERT was designed for transfer learning, where it was first pretrained on 2 unsupervised tasks, that is, masked language modeling (MLM) and next sentence prediction (NSP), before fine-tuning for specific downstream applications such as text classification [29].

Similar to BERT, both CharacterBERT_medical and BERT_medical were pretrained on the MLM and NSP tasks, but using data from both general and biomedical domains [28]. The general-domain corpus included 5.99 million Wikipedia documents and 1.56 million OpenWebText documents, whereas the biomedical-domain corpus included 2.09 million clinical notes from the MIMIC-III (Medical Information Mart for Intensive Care III) database [32] and 2.33 million abstracts from PMC Open Access [33]. Unlike BERT, CharacterBERT_medical does not embed tokens (words or subwords) directly. Instead, it represents each character as a 16-dimensional real-valued vector and represents each token as a sequence of character embeddings. The character embedding sequence is passed through multiple parallel 1-dimensional convolutional neural networks (CNNs) with varying filter sizes. The outputs of these CNNs are max-pooled along the character dimension and concatenated to produce a unified representation for the sequence. The unified CNN output is then passed through 2 highway layers (which combine nonlinear transformations with residual connections) [34] and projected into a 768-dimensional space, matching the size of BERT’s word piece representation. An evaluation on medical text classification tasks, including the detection of chemical-protein interactions (ChemProt [35]) and drug-drug interactions (DDI [36]) from free text, showed that CharacterBERT_medical and BERT_medical consistently outperformed the original BERT model [28].

For this study, we developed CharBERTDrug and BERTDrug by fine-tuning CharacterBERT_medical and BERT_medical on RxNorm-augmented data (see section Model Development and Validation Using RxNorm-Augmented Data). We included BERTDrug in this study despite the primarily noncontextual nature of the misspelling detection task for two main reasons. First, many drug names are multiword expressions (eg, abacavir-dolutegravir-lamivudine and Admelog SoloStar U-100 Insulin), and BERT-based models are better able to leverage this local lexical context than noncontextual methods. Second, BERT models use subword tokenization, which, similar to character n-gram approaches, can help identify plausible word structure even when terms contain rare or atypical character sequences.

A total of 69,824 RxNorm drug names were used for model development and internal validation, which contained 41,941 (60.1%) generic names and 27,883 (39.9%) branded names (Table 1). On average, an RxNorm drug name contains 2.4 (SD 1.5) words and 19.0 (SD 12.1) characters, with generic drug names longer than branded names (Table 1). The length of drug names is similar across the training, development, and test sets.

A total of 3803 LTCDC drug names (see Table S1 in Multimedia Appendix 2 for examples) were used for external validation of the misspelling detection models, which contained 1665 (43.8%) low-frequency names and 2138 (56.2%) high-frequency names (Table 2). Analysis of 690 misspelled LTCDC terms (see Table S2 in Multimedia Appendix 2) showed that most misspellings could be generated by our typo-creation method. Misspellings involving the exchange of 2 nonadjacent characters and phonetic-like substitutions, which could in principle be generated by our method but likely with low frequency, together accounted for approximately 10% of errors.

With oversampling applied to the training set and performance evaluated on the development set (see Multimedia Appendix 3: Table S1 for BERTDrug and Table S2 for CharBERTDrug), BERTDrug’s performance plateaued at an oversampling rate of 6 (F₁-score=0.863; ROC-AUC=0.950; PR-AUC=0.952), while CharBERTDrug plateaued at a rate of 4 (F₁-score=0.838; ROC-AUC=0.910; PR-AUC=0.907). Overall, BERTDrug outperformed CharBERTDrug across all evaluation metrics and oversampling rates.

When evaluated on the test set (Table 3), BERTDrug₆ (model trained at an oversampling rate of 6) performed best (F₁-score=0.859; ROC-AUC=0.947; PR-AUC=0.948), followed by CharBERTDrug₄ (F₁-score=0.833; ROC-AUC=0.906; PR-AUC=0.900) and fastText_ML (F₁-score=0.795; ROC-AUC=0.846; PR-AUC=0.837). SpellChecker achieved a high recall (0.941) but the lowest precision (0.689). fastText_ML achieved the highest precision (0.814). Resampling-based evaluation showed that model performance remained stable and showed similar patterns of performance differences between models. The 2-way bootstrap analysis showed statistically significant differences (ie, with 95% CIs for model differences excluding 0) between each transformer model and each baseline model across most metrics. The only nonsignificant comparison was between BERTDrug and SpellChecker for recall (95% CI for difference –0.005 to 0.005). Evaluation stratified by term type showed that branded drug names were more challenging for all models (Table 3). Table S3 in Multimedia Appendix 3 provides the full results for all evaluation metrics.

On the cleaned out-of-vocabulary LTCDC dataset (first block in Table 4 and Figure 2), CharBERTDrug achieved the best F₁-score (0.696) and ROC-AUC (0.788), while BERTDrug achieved the best PR-AUC (0.665). SpellChecker achieved the highest recall (0.996). Full results for all evaluation metrics are provided in Table S1 in Multimedia Appendix 4. Bootstrap analyses of model differences showed statistically significant differences between the best-performing transformer-based model and each baseline model across most metrics, with 95% CIs of model differences excluding 0 (Table 4, note c). In contrast, differences between the two transformer-based models were not statistically significant for most metrics (Table 4, note d). Detailed model comparison results are provided in Multimedia Appendix 6. The ROC curves also demonstrated clear performance differences among the models (Figure 3). Evaluation on the cleaned full LTCDC dataset (fourth block in Table 4) and the uncleaned LTCDC dataset (Table S2 in Multimedia Appendix 4) showed similar patterns in model performance differences.

Evaluation stratified by term type showed that branded drug names were more challenging for all models (Table 4 and Table S1 in Multimedia Appendix 4). CharBERTDrug performed best on most metrics for generic drug names in both the cleaned out-of-vocabulary set and the cleaned full set (second and fifth blocks in Table 4), whereas BERTDrug performed best on most metrics for branded drug names in both the cleaned out-of-vocabulary set and the cleaned full set (third and sixth blocks in Table 4). Bootstrap analyses of model differences showed that, for branded drug names, the two transformer-based models did not differ significantly on most metrics (Table 4, note d).

Frequency-stratified evaluation (Tables S3 and S4 in Multimedia Appendix 4) showed that CharBERTDrug performed best on high-frequency terms, while BERTDrug performed best on low-frequency terms across most metrics. The high-frequency subsets were extremely imbalanced, with no positive instances in the cleaned LTCDC dataset and only 9 (0.4% of 2138) positive instances in the uncleaned LTCDC dataset. In the uncleaned dataset, metrics sensitive to minority-class performance, including precision, F₁-score, and PR-AUC, were poor across all models (eg, F₁-score≤0.05), whereas accuracy and ROC-AUC were comparable to those observed for low-frequency terms (Table S4 in Multimedia Appendix 4).

A total of 3586 cleaned LTCDC terms were analyzed. Among them, 1088 (30.3%) terms were classified as easy (ie, consistently predicted correctly by all models), while 100 (2.8%) terms were difficult (ie, consistently predicted incorrectly by all models). Among the 100 difficult terms, only one term (1%), Cyanocobalamine instead of Cyanocobalamin, was a misspelled drug name that was incorrectly flagged as correct by the models. The remaining 99 (99%) difficult terms were correctly spelled but incorrectly flagged as misspellings by the models; among them, 73 (73.7%) were branded drug names, including Hempvana (topical product for pain relief), Geri-kot (senna), Veozah (Fezolinetant), Zenoptiq (ocular nutritional supplement), and more than 20 branded vaccine names (eg, Flublok Quad 2019-2020 (PF) (flu vac qv 2019(18yr up)rc(pf)) and Moderna COVID-19 Bival Booster).

Among the 128 terms that were predicted correctly by SpellChecker but incorrectly by both BERTDrug and CharBERTDrug, 72 (56.3%) were misspelled. In contrast, all 382 terms that were predicted incorrectly by SpellChecker but correctly by both BERTDrug and CharBERTDrug were correctly spelled medication names.

As shown in Multimedia Appendix 7, in the RxNorm test set, after adjusting for term length, BERTDrug was less likely to make correct predictions for branded drug names than for generic names (odds ratio [OR] 0.883, 95% CI 0.825-0.945); while CharBERTDrug’s prediction accuracy did not differ significantly by term type (OR 0.989, 95% CI 0.928-1.054). In the LTCDC out-of-vocabulary test set, after adjusting for term length, both BERTDrug and CharBERTDrug were less likely to make correct predictions for branded drug names than for generic names (BERTDrug: OR 0.528, 95% CI 0.423-0.660; CharBERTDrug: OR 0.353, 95% CI 0.280-0.444).

Term length was positively associated with prediction accuracy in the RxNorm test set but negatively associated with prediction accuracy in the LTCDC test set (Table S1 in Multimedia Appendix 7).

The GPT-4o model (baseline prompt + system role, temperature=0.0) showed strong stability and good overall performance (Table S2 in Multimedia Appendix 5) and was therefore included in the formal evaluation. Both specialized transformer-based models (ie, BERTDrug and CharBERTDrug) outperformed GPT-4o on RxNorm terms across most metrics, including F₁-score (BERTDrug: 0.855, CharBERTDrug: 0.831, GPT-4o: 0.721), ROC-AUC (BERTDrug: 0.951, CharBERTDrug: 0.911, GPT-4o: 0.856), and PR-AUC (BERTDrug: 0.953, CharBERTDrug: 0.908, GPT-4o: 0.838). However, this performance advantage diminished substantially for the LTCDC terms: the transformer-based models outperformed GPT-4o only in precision and specificity, while GPT-4o performed better on all other metrics. Table S3 in Multimedia Appendix 5 provides the full results.

This study presents a deep-learning approach that leverages specialized, transformer-based language models to detect typographical errors in out-of-vocabulary medication names. Internal validation on the RxNorm-derived test set showed strong performance of both BERTDrug and CharBERTDrug models, with ROC-AUC scores of 0.947 and 0.906, respectively. External validation on the LTCDC terms demonstrated adequate generalizability of both models, with ROC-AUC scores of 0.786 for BERTDrug and 0.788 for CharBERTDrug on the cleaned out-of-vocabulary set and 0.869 and 0.868, respectively, on the cleaned full set.

Detecting misspellings from medical text has been an active area of research. Most prior studies used dictionary-based methods to detect misspelled medical terms [2,4,6,7,15,21,37]. Two recent studies have applied deep learning techniques to correct misspellings in non-English clinical texts; however, neither addressed out-of-vocabulary terms nor independently evaluated their error detection components [22,23]. To our knowledge, this study is the first to develop and evaluate deep learning models—particularly transformer-based language models—that detect misspellings from out-of-vocabulary medication names. Our approach addresses a key limitation of dictionary-based methods, which classify all out-of-vocabulary terms as misspellings and therefore suffer from excessive false positives (ie, low precision) and low specificity. For example, compared to BERTDrug, the dictionary-based SpellChecker had a significantly lower precision (0.486 vs 0.631, 95% CI for difference_{BERTDrug_vs_SpellChecker}: 0.121 to 0.168) and specificity (0.304 vs 0.724, 95% CI for difference_{BERTDrug_vs_SpellChecker}: 0.389 to 0.451) in detecting errors in out-of-vocabulary LTCDC medication names (refer to the highlighted rows in worksheet “Table4_Appendix4-TableS1” in Multimedia Appendix 6). This means that for every 100 correctly spelled new drug names (ie, out-of-vocabulary terms), SpellChecker generates 42 more false alarms for misspellings than BERTDrug (100 * (0.724 – 0.304)). Expanding the dictionary used by SpellChecker can improve precision and specificity, but will decrease dictionary search efficiency. More importantly, it requires ongoing monitoring of newly developed drugs and continual updates to the dictionary. In contrast, the transformer-based models (BERTDrug and CharBERTDrug) provided a superior balance across precision, specificity, and recall, along with improved overall performance as measured by F₁-score and ROC-AUC. This finding highlights the advantage of using transformer-based language models for detecting misspellings in out-of-vocabulary terms. It is worth noting that even in clinical care settings—where high sensitivity in medication error detection is prioritized—a balance across precision, specificity, and recall is still crucial, particularly for hospitals that intend to deploy fully automated pipelines for error detection and correction. Excessive false alarms during error detection not only increase alert burden but may also prompt unnecessary downstream corrections, inadvertently creating new medication name errors.

Compared to the medical domain, progress in misspelling detection and correction has advanced more rapidly in the general domain. Recent research explored end-to-end approaches that leverage deep learning-based sequential models to jointly detect and correct misspellings [42-46]. Commonly used techniques included recurrent neural networks (RNNs) such as long short-term memory (LSTM) and gated recurrent unit (GRU) models [43,45], as well as transformer-based models like BERT [42,47]. Our approach is closely related to studies that used BERT-based models for error detection [44,47], but it differs in model implementation. Jayanthi and colleagues [44] applied a BERT-based model to detect and correct misspelled words. For each word in the input sequence, their model outputs a probability distribution over a predefined vocabulary. As such, their model could not handle misspellings in out-of-vocabulary words. Klemen and colleagues [47] fine-tuned the Slovene BERT model—SloBERTa [48]—for detecting misspelled words in an input sentence. They introduced a special token after each word (or after the last subword of a word) to facilitate word-level error detection. In contrast, our BERT-based models took a medication name (which was a single-word or multiple-word expression) as the input and classified it as misspelled or correctly spelled. A few prior studies have used character-based CNNs to learn local character patterns and have integrated them with RNNs to detect misspellings [44,45]. Similarly, our CharBERTDrug model combined a character-level CNN with BERT.

On the cleaned full LTCDC set, CharBERTDrug achieved higher recall than BERTDrug (0.780 vs 0.718), while showing slightly lower specificity (0.842 vs 0.854). The difference likely arose from their tokenization focus: CharBERTDrug leverages character-level information, which enhances its sensitivity to subtle misspellings and boosts recall. There is a trade-off between maximizing error detection and minimizing false positives; the choice of model should be guided by the specific needs of the applications. For example, high recall in detecting misspelled medication names is preferred in applications that support pharmacovigilance and safety monitoring, whereas high precision and specificity are more desirable in real-time alerts embedded within clinical workflows, where avoiding alert fatigue is critical—particularly when late-stage quality assurance processes are available to catch additional misspellings.

Both BERTDrug and CharBERTDrug demonstrated significantly higher precision and specificity than the 3 baseline models. As shown in Figure S1 in Multimedia Appendix 8, both models had a substantially lower projected review burden per 1000 medication entries than the baseline models across plausible misspelling prevalences. As expected, SpellChecker had the lowest projected number of missed misspellings (Figure S2 in Multimedia Appendix 8), but it also incurred a substantially higher review burden than BERTDrug and CharBERTDrug (Figure S1 in Multimedia Appendix 8).

Error analysis revealed that the majority of difficult cases for all models were branded drug names that were correctly spelled but misclassified as misspellings. Association analyses also showed that the transformer-based models were less likely to make correct predictions on branded names (Multimedia Appendix 7). A possible reason is that brand names exhibit less regular spelling patterns and are therefore more difficult for the models to learn and recognize. Specifically, unlike generic drug names, brand names lack consistent morphological structure including shared prefixes or suffixes (eg, -statin, -caine, and -olol), which limits the model’s ability to generalize via subword patterns or character-level n-grams and forces the model to rely more heavily on memorization of individual lexical items. In addition, brand names often violate common orthographic and phonotactic patterns of natural language and contain atypical character sequences (eg, Vraylar, Xeljanz, and Qulipta) that are more likely to be interpreted by the model as noise or misspellings.

In addition to term type, term length was also associated with prediction accuracy for the two transformer-based models. However, the direction of this association differed across datasets: longer RxNorm terms were more likely to be classified correctly, whereas longer LTCDC terms were more difficult for the models to classify correctly (Multimedia Appendix 7). One possible explanation is that, although longer terms may provide additional contextual information that can benefit transformer-based models, those in the external evaluation set often contained previously unseen spelling patterns arising from combinations of formulation, strength, dosage form, and abbreviations not represented in the training data.

We also noted that, because only 9 positive instances were identified among the 2138 uncleaned high-frequency terms and all 9 positive instances were nonstandard spellings, all models performed poorly on metrics sensitive to minority-class performance (eg, precision, F₁-score, and PR-AUC) on this subset. The observed pattern of low precision and moderate recall among the transformer-based models suggests that fine-tuning on nonstandard misspellings is needed to improve detection of this type of misspelling. For fully automated classification, future work could also explore hybrid strategies, such as fine-tuning separate models for high-frequency terms, when a sufficient number of positive cases are available for training, and for low-frequency terms.

Generative LLMs have demonstrated strong general-purpose capabilities, including interpreting free-text inputs and generating clinically relevant insights from vast medical knowledge [49,50]. Unlike specialized language models fine-tuned for classification tasks (eg, the models developed in this study), generative LLMs are not inherently designed as probabilistic classifiers. However, they have demonstrated strong performance on complex clinical classification tasks such as disease diagnosis [51,52]. In our secondary analysis, we found that our models outperformed the generative LLM GPT-4o in precision and speed when detecting misspelled medication names, but with lower overall classification performance than GPT-4o on LTCDC terms. These findings suggest that domain shift can substantially reduce the performance advantages of domain-specific training. Future work should improve model generalizability by incorporating more diverse training data representing the terminology, formatting conventions, and spelling patterns encountered across real-world clinical settings. Note that our analysis is preliminary, and the performance of generative LLMs could potentially be improved with more advanced models and refined prompting strategies. Future studies that systematically compare general-purpose generative LLMs and specialized language models for misspelling detection are warranted to gain deeper insights into their relative strengths and limitations.

This study has several limitations. First, we trained the misspelling detection models using an augmented dataset derived from RxNorm drug names, in which misspellings were introduced through commonly used text perturbation techniques. Although this approach provided a large volume of training data and improved the generalizability of the model, the generated misspellings may not fully reflect the error patterns observed in real-world databases such as LTCDC. This discrepancy likely contributed to the reduced model performance on the external validation set. Future work that incorporates misspelled medication names from real-world datasets into model training may further enhance performance. Second, the generalizability of our findings beyond the LTCDC remains uncertain. Future studies should assess performance in other contexts, such as hospital and retail pharmacy settings, where error patterns may differ. Third, our error analysis revealed that drug brand names were the most challenging cases across all models. Moreover, all models performed poorly in detecting rare misspellings (0.4%) among high-frequency drug names. Because all misspelled high-frequency terms were nonstandard spellings, this low performance is likely attributable to two factors: (1) severe class imbalance and (2) the difficulty of detecting nonstandard spellings. Specialized handling strategies or additional training data may be needed to mitigate these limitations. Fourth, the definition of high-frequency medication terms in the LTCDC external validation set was not fully consistent. LTCDC terms were sampled from two different database releases using different approaches, resulting in two separate criteria for high-frequency terms (>1000 occurrences vs >100 patients). Because earlier database releases were no longer available after LTCDC’s platform transition in late 2024, we could not repeat the original sampling process to harmonize these definitions. This inconsistency may have introduced some heterogeneity into the external evaluation. Fifth, our data augmentation strategy relied primarily on structural text perturbations (eg, insertion, deletion, and substitution). While these transformations capture many typographical errors, they do not explicitly model phonetic misspellings or improper word splitting, which are also common in real-world medication name errors. This may have limited the range of misspelling patterns represented during training and contributed to the reduced performance on the external validation set. Future work should incorporate phonetic and segmentation-based augmentation strategies, such as dropping silent-like letters, substituting similar-sounding letter groups (eg, ph versus f, c versus k), and inserting or removing spaces or hyphens in long medication names, to generate more realistic misspellings in the training data. Sixth, LTCDC terms frequently combine drug formulation, strength, dosage form, and abbreviations, making them structurally different from RxNorm drug names. This compositional shift may confound the observed association between term length and model performance in this dataset. Seventh, many medication names in the RxNorm and LTCDC datasets, or their component words, are also available through publicly accessible online resources and may have been included in the pretraining corpus of GPT-4o. Therefore, benchmarking GPT-4o on terms sampled from these datasets may introduce potential data contamination or data leakage, which could lead to an overestimation of model performance. Finally, we did not examine misspelling detection in longer clinical contexts because appropriate annotated datasets for model training and evaluation were not available. Extending this work to detect misspelled drug names within richer clinical context would be an important and practically relevant direction for future research.

Our findings demonstrate that specialized language models can improve the accuracy of automated drug-name misspelling detection, particularly for out-of-vocabulary terms. When paired with downstream error correction methods, these models may further enhance medication data quality and, in future applications, support patient safety.