The rise of health 2.0 has significantly increased the consumption of health-related information on social media in recent years. Worldwide, 34.7% of adult internet users frequently search for health-related information online, with approximately 58.5% of adults in the United States specifically engaging in such searches as of 2023 [1,2]. Analyzing online discourse on health-related topics offers valuable insights into the lived experiences and perspectives of affected individuals [3-5]. The critical findings of these analyses are particularly significant for understanding complex medical conditions, such as different types of cancer, cognitive and mental health conditions, autoimmune disorders, and metabolic and endocrinological conditions. These medical conditions often manifest with diverse symptoms that vary significantly among patients [6-8]. Furthermore, there is often no consensus among established medical sources regarding the full range of possible symptoms for such conditions [9,10]. This variability poses challenges for patients, health care providers, and researchers seeking a comprehensive understanding of these symptoms and their characteristics. Thus, the study and analysis of symptoms associated with a disease, referred to as symptomatology, is crucial for understanding complex diseases [11]. We argue that a robust approach to extracting disease-specific symptoms from individuals’ lived experiences shared on social media can address uncertainties surrounding complex medical conditions. This approach would also allow for the identification of symptoms that patients associate with a condition (eg, brain fog with post–COVID-19 condition), prevalence of symptom co-occurrence (eg, unexplained weight loss and pain with cancer), and how symptoms vary based on patients’ comorbidities and preexisting conditions.

Recent advancements in natural language processing (NLP) have enabled public health researchers to explore medical information extraction tools [12-15] and develop novel techniques [16-18] for analyzing online health discourse, including patient-reported data from social media. However, off-the-shelf medical information extraction tools primarily excel in extracting well-known medical entities (eg, diseases, drugs, and basic symptoms). Their limitations become evident with granular, patient-centric details (eg, unusual symptom descriptions, implicit context, or subtle adverse events), which often require specialized rules or retrained models for accurate detection [19-22]. Modern transformer-based models used with medical named entity recognition have shown promising results in symptom extraction [23,24]. However, these models typically require large amounts of domain-specific data and are mostly trained on clinical notes (electronic health records), which limits their effectiveness when dealing with colloquial health discourse generated by patients [7,23-25]. A family of bidirectional encoder representations from transformers (BERT)–based models fine-tuned on colloquial data markedly improves symptom recognition in informal texts; nevertheless, their error analysis indicates that the best-trained models still capture only approximately 25% to 50% of lengthy, figurative symptom phrases and continue to misclassify many subtle or previously unseen expressions (eg, “mild difficulty taking deep breaths” or “chest again felt a little tight”) [26]. A recent NLP pipeline [27] used to extract symptoms from patient narratives on social media proposed a dual-corpus symptom lexicon containing keywords (eg, fever and cough) and related modifiers (eg, mild or severe) followed by applying a rule-based postprocessing step to merge the 2. However, this fixed rule-based approach may struggle to report symptoms whenever people describe them implicitly, with spelling noise, or in new phrasing that never entered the seed lists (eg, “can’t breathe properly” or “my lungs felt on fire”).

While the existing lexicon-based approaches have successfully used predefined resources and deep learning frameworks for broad public health surveillance, their reliance on explicit wording limits their ability to capture the full range of informal expressions. On the other hand, large language models (LLMs) have significantly accelerated medical NLP research, especially in the field of information extraction [16,17,21,28]. While some studies have demonstrated the potential of LLMs for health information extraction from social media [29,30], they face challenges when extracting specific contexts such as health advice and disease-specific treatment [12,16].

To overcome the aforementioned limitations of symptom extraction from social media discourse, we propose a novel NLP pipeline that systematically combines the capabilities of LLMs and other NLP methods with feedback from domain experts, resulting in a scalable and practical solution to the problem of symptom extraction from patient-reported social media discourse.

While the proposed pipeline can be applied to explore a plethora of medical conditions for which there is an abundance of patient-reported online discourse, we focused on a common chronic condition with significant uncertainty, namely, polycystic ovary syndrome (PCOS), as a use case in this paper. According to the World Health Organization [31], an estimated 116 million women (3.4% of reproductive-age women) worldwide are affected by PCOS, and its prevalence is on the rise. Despite its widespread impact, the full spectrum of PCOS remains insufficiently explored [32]. Patients report a diverse range of symptoms that vary in severity depending on previous medical history and their effect on daily life. The combination of varied symptoms, insufficient treatment guidelines, and instances of medical gaslighting often leads to psychological distress, prompting many individuals to seek support and share experiences on social media platforms [33], including Reddit. Existing research leveraging social media data to study PCOS is still underdeveloped [4,34-36]. This creates a unique opportunity to systematically identify self-reported symptoms and related symptomatology based on the lived experiences of thousands of individuals and, thereby, inform the clinical understanding of this complex medical condition. We make 3 primary contributions. First, we developed a novel lexicon-based symptom extraction (LSE) method to identify diverse disease-specific symptoms from individuals’ experiences shared on social media. This method can effectively detect even subtle symptoms mentioned in user discourse. We applied our LSE method to a novel PCOS subreddit dataset, achieving statistically significant improvements over state-of-the-art medical NLP baselines.

Second, we conducted an in-depth analysis of peer discussion threads on the extracted PCOS symptoms to reveal a set of lesser-known yet prevalent symptoms that individuals associate with PCOS and associated co-occurring symptoms and comorbidities. These findings can contribute to clinical research and public health research in a community-informed way.

Third, we curated a novel Reddit dataset containing 59,962 and 641,441 relevant posts and comments, respectively. This dataset captures self-reported PCOS symptoms and their context from 68,010 affected individuals. In addition, we curated a human-annotated, labeled dataset of 1000 posts of patient-reported symptoms related to PCOS based on guidance from domain experts and patient input. We release these resources in a public domain. These unique, rich resources can enable medical NLP and computational health researchers to further investigate the symptomatology of complex medical conditions identified and analyzed online.

We selected Reddit as a source of data because of its anonymous nature. It encourages users to share personal experiences on stigmatized topics [4,37]. The data collection process began with the identification of the r/PCOS subreddit by considering the group posts’ availability and popularity among 153,000 members engaging in discussions on PCOS. We systematically collected data from this subreddit using the Python Reddit API Wrapper package (Python Software Foundation) and the Pushshift application programming interface (API) [38,39]. We obtained all the posts and comments from the r/PCOS subreddit from August 4, 2010, to December 31, 2022. This timeline was chosen to ensure a comprehensive capture of recent as well as previous discussions. For each post, we retained the date, flair name, unique post ID, post title, post content, score, upvotes, and comments due to their utility and relevance.

During preprocessing, we discarded entries with deleted posts or those in which the post content had <5 words as the brevity poses challenges to deriving meaningful insights from posts. We also removed posts containing irrelevant text (eg, polls, survey links only, or URLs). The URL-heavy posts were classified using a rule-based heuristic combining personal pronoun frequency and URL-to-text ratio. We provide a summary of the steps to remove irrelevant text from social media data in Table S1 in Multimedia Appendix 1. This filtering process enabled us to focus on patient-authored, experience-rich discussions. Finally, we obtained a primary dataset comprising 59,962 posts and 641,441 comments by 68,010 active unique users. Identifying task-relevant data is crucial for data-driven approaches [12] as it significantly impacts the efficiency of downstream tasks. We leveraged Reddit’s post “flair” option to filter relevant data. On Reddit, a post flair is a customizable text or icon that appears next to a post title to provide additional context or categorization, similar to hashtags available on other platforms. We reviewed 30 randomly selected posts from several flairs of the PCOS subreddit. We identified that, among all the flairs, posts in the “General/Advice” flair were more likely to share knowledge and seek advice on symptoms. Further investigation on the most frequent unigrams of these flairs supported our qualitative analysis, where the presence of symptom-related unigrams such as “period,” “symptom,” “doctor,” “birth,” “control,” and “diagnosed” was predominantly observed in the “General/Advice” flair. The distribution of frequent unigrams in the “General/Advice” flair is shown in Figure S1 in Multimedia Appendix 1. On the basis of these findings, we chose the “General/Advice” flair for extracting a wide range of symptoms. In addition, the “General/Advice” flair is more popular in the community as it contains the highest number of posts, as shown in Table S2 in Multimedia Appendix 1. This selection ensured a broad range of symptoms and valuable insights. Finally, we developed a corpus of 10,500 posts for the symptom identification task, randomly selected from the posts with the “General/Advice” flair.

We propose an LSE method to analyze self-reported symptoms of a complex medical condition using social media discourse. Our proposed method consists of three main steps: (1) symptom-related lexicon extraction, (2) symptom identification, and (3) symptom normalization, as illustrated in Figure 1.

We acknowledge that the research on symptom extraction from social media has multifaceted ethical concerns and requires careful attention to privacy, consent, and the responsible use of data and technology. The research was approved by Bangladesh University of Engineering and Technology's Institutional Review Board. Although Reddit is an anonymous platform, we ensured that no personally identifiable information was inadvertently revealed during data collection, analysis, or publication. All Reddit posts and comments cited as examples in this paper were paraphrased to protect user privacy. We also recognize that the demographic composition of Reddit users might not be fully representative of the broader population with PCOS, potentially limiting generalizability. Therefore, findings derived from Reddit discourse should be generalized cautiously to other populations or settings. In addition, we recognize that any biases in the dataset and the proposed model are unintentional. As the outcomes of this work are based on social media content, we emphasize the superiority of medical evidence–based facts in cases of conflict, particularly concerning direct clinical practice and medical diagnosis. We strongly encourage the use of the model for research and exploratory purposes, serving as a foundation for further developments in computational health research.

We initially separated 1000 sample posts of various lengths from our dataset for results evaluation. Considering the influence of post length on symptom identification, we categorized the posts into three groups: (1) short posts of <100 words, (2) medium-length posts of between 100 and 300 words, and (3) long posts of >300 words. Post length distribution in the dataset revealed that 45% (450/1000) of the posts belonged to the medium-length category (Table S8 in Multimedia Appendix 1). To ensure a balance across all categories, we selected 500 posts from the medium-length category and 250 posts from each of the short and long post categories for curating test data.

We recruited a medical graduate with domain knowledge and a patient with PCOS lived experience to perform annotation. Before annotation, both annotators participated in progressive training sessions and interactive discussions with medical specialists to refine their understanding and improve annotation consistency. During this phase, the medical specialist primarily identified 3 potential disagreements between the annotators. These observed disagreements were instances in which (1) the medical graduate often used clinical terminology (eg, “amenorrhea”) whereas the patient used informal language (eg, “no period for months”); (2) the patient included self-reported experiences (eg, “hypothyroidism”) that lacked clinical validation whereas the clinician strictly adhered to established PCOS diagnostic criteria; and (3) the medical graduate inferred implicit symptoms (eg, laboratory test results) from domain knowledge, which was lacking in the patient annotation. Given our goal to prioritize the capture of patient-experienced symptoms, medical specialists suggested resolving these discrepancies by (1) avoiding overly technical clinical terms unless they were explicitly mentioned, (2) retaining self-reported experiences unless they were clinically invalid, and (3) including implicit symptoms as long as they were related to PCOS. Following these guidelines, both annotators extracted symptoms independently. We adopted the combination of both annotators’ symptom lists per post to maximize coverage and ensure the consistency and reliability of the ground truth dataset.

In our ground truth annotation, annotators were not restricted to predefined symptom lists. They identified any symptoms that they perceived as PCOS related, resulting in free-text labels. Moreover, each post included multiple symptoms, making the traditional Cohen κ not suitable. Instead, we evaluated interannotator agreement using relaxed-match F₁-score due to the generative nature of our annotation [16,54,55]. We used BioBERT-based semantic similarity with the medical graduate’s annotation as a reference. We achieved an F₁-score of 0.82, with a recall of 0.84 at a similarity threshold of 0.6, indicating substantial agreement considering the realistic complexity inherent in patient-derived, generative-nature symptom annotation.

LSE resulted in a comprehensive list of 64 unique PCOS symptoms (Table 2) from 10,500 user posts. We also determined the unique users reporting each symptom in our primary dataset, with counts ranging from 31 to 16,001. This confirmed that LSE captures widely experienced symptoms, whereas less common symptoms are not ignored. In addition, this list provides a wide range of coverage as we cross-checked it with 7 evidence-based consumer health websites: the World Health Organization [31]; the Mayo Clinic [56]; WebMD [57]; the International Classification of Diseases, 10th Revision [58]; Johns Hopkins Medicine [59]; the National Health Service [60]; and the Centers for Disease Control and Prevention [61]. All symptoms mentioned in these sources were included in our list except for 4: sleep apnea, high blood pressure, low progesterone, and dandruff. Upon examining the clusters, we found that these 4 symptoms were represented within their respective key clusters—insomnia, high cholesterol, low estrogen, and high DHEA—due to their close medical relevance. Furthermore, all the forums recognized the 10 most frequently reported symptoms in our dataset (Table S9 in Multimedia Appendix 1). On the other hand, we identified 28 symptoms not acknowledged by any of the eHealth forums that we mention as potentially emerging symptoms in the Discussion section. Thus, our comprehensive symptom list reflects the diversity of self-reported PCOS lived experience symptoms and aligns closely with authoritative medical sources, helping standardize the variability in the symptom descriptions.

Despite the promising results, this study has several limitations. First, our study focused on a single health condition and a single social media platform, which may introduce platform-specific biases and limit generalizability. Second, both the lexicon and the evaluation sample sizes are relatively small for broader conclusions. Third, our findings are exploratory and hypothesis generating and not yet clinically validated.

We also acknowledge specific implementation limitations. Although the LSE pipeline is designed for modularity and adaptability, applying it to other complex health conditions may require empirical adjustments. Determining optimal clusters for symptom identification and establishing semantic mapping rules for symptom normalization would require condition-specific customization. In this study, we used silhouette scores and inertia analysis to determine the optimal cluster metrics, such as the number of clusters (k). However, we recognize the limitation of distance-based metrics due to the curse of dimensionality, which may not fully capture cluster shape or density nuances. Despite this, our bootstrap stability analysis demonstrated consistently high Adjusted Rand Index (ARI) values across various k settings, indicating that our symptom clusters were robust to moderate changes in the number of clusters and to the random removal of data points. The performance of this framework may vary depending on the extracted lexicon due to linguistic variability and the nature of symptom discourse. Future work should focus on validating this method across diverse conditions, platforms, and languages. In addition, we plan to conduct online surveys and focus groups involving both patients and practitioners to assess the clinical relevance of our findings.

This work focused on extracting and analyzing disease-specific symptoms from individuals’ lived experiences with an application to the PCOS use case. The uniqueness of our work lies in several key aspects. The extraction of a symptom-related lexicon ensures the capture of subtle mentions without overlooking informal expressions on social media. We generated disease-specific symptoms by integrating LLMs into the NLP pipeline rather than relying on traditional NLP methods, which often obtain generic symptoms. In addition, analyzing PCOS symptomatology across varied dimensions revealed unique findings, such as emerging new symptoms expressed in patient-centric language, symptom co-occurrence patterns, and associated comorbidities. The outcomes of this work may help eliminate uncertainties surrounding complex medical conditions and contribute to clinical research and public health policies in a community-informed way.