Background: Previous research shows that being a “digital native” or growing up in a digital environment does not necessarily lead to increased digital competencies, such as digital health literacy or evaluation of webpage quality.
Objective: This study showed how medical students searched for health information online, specifically the recommended testing for histamine intolerance, by comparing the use of various search engines (Google, Medisuch, and a website of the student’s choice) to find out more about search strategies in future health professionals. As Medisuch presents a qualitatively better search engine, we assumed that medical students using this search engine might find valid information faster on more reliable webpages, and might recommend the correct diagnostic steps for histamine intolerance to their patients more often than students using a generic search engine like Google.
Methods: Medical students in their third year of study were asked to find the relevant diagnostic steps of histamine intolerance online. They were randomly assigned to use one search engine: Google, their personal choice, or Medisuch. Their process of seeking information online was video recorded.
Results: In total, 140 medical students participated in this study. The total number of webpages found did not differ among the groups (P=.52). Students using Medisuch (P=.02) correctly identified the elimination diet as a relevant diagnostic step more frequently. The provocation test was reported by almost half of the students independent of the search engine used. In general, medical students commonly identified trustworthy webpages in all three groups (Google: 36/44, 82%; free choice: 31/36; 86%; and Medisuch: 35/45, 78%).
Conclusions: The results indicate that medical students were able to find trustworthy health-related information online independent of the search engine used. Medical students that are digital natives seem to have proper internet skills and a knowledge of how to use them. They entered specific medical terms (evidence-based diagnostic steps) or names of reliable webpages (DocCheck) in the search engines to gain correct information. However, it remains to be seen if this behavior can be called true “digital literacy”.
The internet is omnipresent and has become the primary source of information for many . It has been reported that 52.2% of Europe’s population has used internet searches for health and health-related issues [ ]. Çoklar et al [ ] reported that online information search strategies represented one of the most important variables in effective and efficient internet use. Previous research has shown that online search strategies might be influenced and explained by the time spent online, Web experience, and individual difference (like domain knowledge or epistemological belief) [ - ]. An investigation of university students’ online search strategies in different contexts indicated that the students were able to search online for daily life information. However, they had difficulties in the online search strategies used for learning activities, and the authors suggested that teachers should help students develop online search strategies for academic uses [ ]. The need for guidance—particularly for people with little expertise—has also been highlighted by Armstrong and Large [ ] in their manual on online search strategies.
Furthermore, it has been shown that falling in the category of “digital natives”  or showing “digital nativity” [ ] (ie, being born in a digital world and, therefore, frequently using internet or media devices since early childhood) does not automatically correlate with online literacy [ , ]. Therefore, attempts have been made to clarify quality indicators for online information.
One such approach involves introducing specific search engines that prefilter for the user and only show reliable results, such as the German webpage Medisuch . Another approach is to certify webpages with trustworthy, evidence-based medical content and introduce certificates that make it easier for average people and professionals to decide quickly if a webpage can be trusted. At first glance, this approach seems easy; however, people seeking information online still have to be properly trained [ ]. Examples of certificates available in the topic of health-related information are the afgis certificate and the Health On the Net Foundation Code of Conduct (HONcode) certificate. For a more comprehensive list, see Pauer et al [ ].
To properly advise patients, medical students need to be generally aware of diagnostic steps; if they lack the knowledge, they should know how to find reliable information. As medical students are still learning, specific training can impact their attitude and behavior. This study compared various search engines to investigate how medical students search for health information online. The aim of this study was to track medical students’ searches for health information online and identify potentially weak strategies, which could be addressed specifically within the medical curriculum. We wanted to compare three different approaches and focus on the medical students’ findings on histamine intolerance. As Medisuch presents a qualitatively better search engine , medical students using it may find valid information faster on more reliable webpages, and may recommend the correct diagnostic steps for histamine intolerance to their patients more often than students using a generic search engine like Google.
Study Design and Participants
This study was performed in an explanatory cross-sectional manner. Third-year students from the Faculty of Medicine of the University Hospital Tuebingen, Germany, were recruited from their curricular courses in the Department of Internal Medicine VI. They had all received theoretical input on functional disorders and differential diagnoses with a focus on intolerances before participating in the study. Group sizes consisted of 8 to 16 students per teaching session. Students were taught by experienced physicians. Participation in the course was mandatory, but participation in the study was on a voluntary basis.
The study received ethical approval from the Ethics Committee of the Tuebingen Medical Faculty (443/2018BO2). All participants gave written informed consent. They did not receive a reimbursement for their participation.
Process of Study
Students of the university must attend a 2-week course on psychosomatic medicine in their third year. Every 2 weeks, a new group of approximately 12 to 15 students start the course. The topic is “somatoform disorders” on the third day of the course. Students received a case report involving a patient who consulted her physician due to abdominal pain. The patient in the fictitious case had already researched her health problem beforehand and asked specifically about potential causes such as problems with digestion of histamines. The patient wanted to know which steps had to be taken to rule out or confirm histamine intolerance. Each student had access to one stationary computer with connection to the internet via Internet Explorer or Firefox (both were installed and students were free to choose). Students were instructed to complete a worksheet about histamine intolerance by searching for related information online. They were randomly assigned to one of three groups: Google , free choice, or Medisuch [ ]. In the free choice group students could choose any search engine they wanted to use. There were no prerequisites for participation, and they were not taught further information about the various search engines. Medisuch is a specific search engine that prefilters the information found and only shows reliable results (ie, medical webpages that are certified or evidence-based) [ ]. The students had 10 minutes to research information online and mark their findings on the worksheet, which was collected by a study assistant after the time period. Teaching then continued with an interactive discussion where students were informed about the correct diagnostic steps as well as quality indicators of good webpages and apps. shows an overview of the study. The quality indicators were based on the checklists used by afgis and HONcode, such as objectivity or accuracy. For a more comprehensive list, see Pauer and colleagues [ ].
Students were asked about demographic variables such as age and gender, as well as experience with electronics such as possession of electronic devices and time spent online per day. They also rated their competence on the topic of histamine intolerance on a 6-point Likert scale (0 = not competent at all and 5 = highly competent).
Before starting the teaching session, students were asked to start Morae (TechSmith, Michigan)  on their computer, which was used to track students’ online search for health information via a screen recording of their desktop. Morae has been previously applied within the research area of online search behavior by university students [ - ]. The software generates quantitative data on different variables such as time spent on each webpage and clicks made. To the best of our knowledge, no other standardized procedure exists that would successfully analyze the video-recorded data in a qualitative way. In alignment with the previously described procedure [ ], we aimed to develop a categorical system (eg, search engine, number of webpages, search strategies) to guarantee standardized analysis and evaluation of the Morae videos.
All Morae videos were coded according to this categorical system by experienced members of the research team. All webpages students had visited were listed alphabetically, and their quality was classified according to a rating sheet by two independent raters. In cases of a disagreement, a third independent rater was consulted.
Students answered specific questions about the search engine(s) used, keywords entered, and the number of webpages they consulted before making their diagnostic recommendation. Additionally, they rated the number of pages considered as trustworthy as well as the number of pages they would recommend to a patient. Trustworthy was defined as an evidence-based and certified webpage; thus, untrustworthy was defined as a nonevidence-based one such as an advertisement or self-made webpage by an individual. To get an idea on the effectiveness of medical students’ searches, the number of pages opened and the keywords needed to get to the desired answer were considered as relevant markers.
Students had to give specific recommendations on the next diagnostic steps. They were asked to choose one or more of the following options: antibody testing, assessment of diaminoxydase, nutritional diary, elimination diet, H2 breathing test, histamine testing, provocation test, and test of urine and feces. To pass, students needed to mark one or more of the following options: provocation test, nutritional diary, or elimination diet, as these were the diagnostic factors recommended by a clinical expert.
Statistical analysis was performed using SPSS 25 (IBM Corp, Armonk, New York). Mean values, associated SDs, frequencies, and percentages of relevant factors such as age and gender were calculated. To test possible relationships among the variables, Chi-squared tests were used. Analysis of variance (ANOVAs) were conducted to compare differences between mean values. In addition, the Pearson correlations were calculated. Beforehand, distribution of data was tested by the Kolmogorov-Smirnov test. P values<.05 were reported as significant.
There were 140 students who took part in the survey, and the response rate was 83.3% (140 /168). The average age was 24.36 (SD 2.84) years. Females constituted 61.7% (85) of the participating students. There were 50 students randomly assigned to Google, 46 to Medisuch, and 44 to the free choice group (χ2278=280.0, P=.46). All participants completed the accompanying worksheet.
There were no significant differences between the groups with regards to age (F2,135=5.04, P=.008), gender (χ24=4.5, P=.34), and previous formal medical or information technology (IT) training (χ22=1.5, P=.23). Smartphones were owned by 78.6% (110) of the students; only 2.1% (3) had a smartwatch; 46.4% (65) had a tablet; and 72.1% (111) had a laptop. The vast majority (131, 93.6%) reported spending from 1 to 6 hours online daily.
Videos were successfully recorded from 125 students (44 from the Google group, 45 from the Medisuch group, and 36 from the free choice group). The remaining 15 recordings were lost due to technical reasons such as the recording or storing not working. The following data are based on these video recordings except those on diagnostic recommendation, which were based on the worksheet.
There was no significant difference between groups regarding the total number of webpages students considered before making their diagnostic choice. The same result was shown for the number of pages regarded as useful. However, students of the free choice group (mean 0.88, SD 0.79) reported significantly fewer pages as recommendable to patients than the other two groups (F2,133=5.04, P=.008; MGoogle 1.55, SD 0.91; MMedisuch 1.52, SD 1.53).
Information seeking-behavior of students regarding the total number of webpages, the number of pages considered useful, and the number of pages considered recommendable for each group, as well as the means and SDs of the webpages are shown in. Students in the free choice group opened significantly fewer recommendable pages (F2,133=5.04, P=.008).
There were no significant differences between groups in regards to the number of keywords entered in the search field or the number of webpages accessed ().
There was no significant difference regarding the amount of keywords and webpage names found by students for each group. Means and the corresponding SDs are shown in.
There was a highly significant difference between groups in whether or not the students entered specific medical terminology in the search engine (χ24=16.6, P=.005). The majority of students in the Google and free choice groups entered either specific medical webpages such as DocCheck or specific medical terminology such as “evidence-based” into the search engine to specify their search; however, in the Medisuch group, more than one-third did not. Of the 44 students in the Google group, 38 (86%) used specific medical terminology for their search. In the free choice group 35 out of 36 (97%) students used specific medical terminology. In the Medisuch group 29 students out of 45 (64%) used specific medical terminology.
Quality of Webpages
In total, 53 different webpages were accessed by the whole sample. Quality ratings showed that 34 pages fulfilled the criteria of a qualified webpage. Interrater-correlation after Cohen d was r=.73.
There were significantly high Pearson correlations between the number of webpages and the number of reliable webpages for all three groups (Google: r=.895; free group: r=.912; Medisuch: r=.860; all P<.001).
There were no significant differences in the frequencies of trustworthy webpages found among the three groups with χ214=16.45, P=.29 The webpage of German national treatment guidelines was used by students of all three groups (Google: 15.9%; free choice: 33.3%; and Medisuch: 51.1%). With regard to the quotient of reliable webpages and all webpages found by students, again, no significant difference was shown (F2,121=1.68, P=.19) between the groups. The mean quotients of the single groups were high with Google: 0.82 (SD 0.20); free group: 0.86 (SD 0.22); and Medisuch: 0.77 (SD 0.23) ().
presents the means (and corresponding SDs) of the quotients of trustworthy or untrustworthy pages by the total number of pages found, which are separated by each group. There were no significant differences for trustworthy and untrustworthy sources.
Independent of the search group, approximately 60% of students identified nutrition diary as a correct diagnostic step (). With regard to the provocation test, almost half the students found this procedure to be correct. The Medisuch student group reported the elimination diet as a correct diagnostic step for histamine intolerance more often than the Google or free choice groups. However, they suggested the wrong answer “antibody detection” significantly more often than the other groups.
|Diagnostic step (correct or wrong)||Google (N=50)||Free choice (N=44)||Medisuch (N=46)||Chi-square (df)|
|Antibody detection (wrong)|
|yes||2 (4%)||1 (2%)||7 (15%)||6.84 (2), P=.03|
|no||48 (96%)||43 (98%)||39 (85%)|
|Histamine testing (wrong)|
|yes||19 (38%)||15 (34%)||13 (28%)||1.03 (2),P=.60|
|no||31 (62%)||29 (66%)||33 (72%)|
|Assessment of diaminoxydase (wrong)|
|yes||29 (58%)||23 (52%)||18 (39%)||3.55 (2),P=.17|
|no||21 (42%)||21 (48%)||28 (61%)|
|H2 breathing test (wrong)|
|yes||0 (0%)||0 (0%)||0 (0%)||N/Aa|
|no||50 (100%)||44 (100%)||46 (100%)|
|Test of urine and feces (wrong)|
|yes||7 (14%)||4 (9%)||7 (15 %)||0.84 (2),P=.66|
|no||43 (86%)||40 (91%)||39 (85%)|
|Nutrition diary (correct)|
|yes||30 (60%)||27 (61%)||28 (61%)||0.02 (2),P=.99|
|no||20 (40%)||17 (39%)||18 (39%)|
|Elimination diet (correct)|
|yes||26 (52%)||19 (43%)||33 (72%)||7.87 (2),P=.02|
|no||24 (48%)||25 (57%)||13 (28%)|
|Provocation test (correct)|
|yes||25 (50%)||21 (48%)||22 (48%)||0.06 (2),P=.97|
|no||25 (50%)||23 (52%)||24 (52%)|
In this explanatory study, we examined medical students’ information-seeking behavior when assigned to use a generic search engine or a search engine of high quality. The hypothesis that students of the qualitatively better search engine would find the correct diagnostic steps for histamine intolerance more often was not fully supported. The Medisuch group significantly identified the elimination diet as a correct diagnostic step more frequently; however, this was not the case for the nutrition diary or the provocation test. The students of the Google and free choice groups reported the antibody detection as wrong more often than the Medisuch group. Furthermore, students of all three groups identified reliable webpages, which indicates that they do have internet skills that allow them to identify reliable information online. They were able to search for information online without navigation issues by entering specific webpages (like Flexicon) or medical terminology (diagnostic steps) in order to find reliable results . Thus, students have successfully avoided the problem of standard search engine strategies, which often produce a multitude of results that users are forced to scroll through and sort the results [ ]. This may be the reason why we did not find any differences between the groups using different search strategies.
A general question arises when viewing these results: Is entering specific terms or webpages into a generic search engine part of “digital literacy”? Many students searched at DocCheck , a webpage that is known for its trustworthy medical information. Students know that this webpage is a reliable source for information, as they use it for their medical studies [ , , ]. Based on these results, medical students were able to comprehend the information found online and develop an understanding of how to find and evaluate it, which could be interpreted as information literacy [ , ]. Furthermore, medical students showed technical use competency by including specific terms in the search strategies [ , ], which previous studies have described as digital nativity [ , ]. Thus, information literacy and digital nativity were shown by the medical students involved in this study, which are determinants of online information search strategies. This point presents an argument for the medical students having digital literacy [ ].
Finding health information online still includes the risk of being misinformed by unreliable information or information providers, which ultimately affects patients [, ]. Consequently, medical students as future health professionals should have appropriate internet skills and use them to help patients find trustworthy information online. Additionally, online technologies for health information should be implemented in the medical curriculum [ ].
One limitation of this study might be that search results and choices of relevant information is only one step out of many. It might be relevant to both track students’ search behavior and assess their cognitive processes preceding the search, such as defining the problem, choosing a certain source of information, or formulating the search strategies . Additionally, in their feedback, students in this study reported that 10 minutes for conducting an information search seemed too short to gather sufficient information. However, as time in daily practice is limited, we consider this a realistic time frame to look for needed information. Furthermore, we need to consider that the number of reliable webpages could differ if other search engines or different topics were used. Generally, it can be argued that the free Web might not offer the same reliable information like customized webpages do (eg, UpToDate or AMBOSS). Future studies, thus, should focus on the patients’ online search strategies and use a similar setting to get more insights into general digital health literacy.
This study showed that medical students are able to search and find relevant medical information online regardless of the search engine used, and thus, this study confirms previous findings of medical students having internet skills in a professional sense. Future studies could focus on how to best integrate these internet skills into the medical curriculum. Furthermore, it needs to be determined if the online behavior of the students involved can be considered proper digital literacy. The next step could also focus on patients and their online search strategies.
We thank all the lecturers of the Department of Psychosomatic Medicine and Psychotherapy. We would also like to thank our study assistant Lisa Maria Wiesner for her help. We also acknowledge the support with financing publication fees by Deutsche Forschungs-gemeinschaft and Open Access Publishing Fund of the University of Tuebingen
Conflicts of Interest
- Kirschner PA, De Bruyckere P. The myths of the digital native and the multitasker. Teaching and Teacher Education 2017 Oct;67:135-142. [CrossRef]
- Kummervold PE, Chronaki CE, Lausen B, Prokosch H, Rasmussen J, Santana S, et al. eHealth trends in Europe 2005-2007: a population-based survey. J Med Internet Res 2008 Nov 17;10(4):e42 [FREE Full text] [CrossRef] [Medline]
- Çoklar AN, Yaman ND, Yurdakul IK. Information literacy and digital nativity as determinants of online information search strategies. Computers in Human Behavior 2017 May;70:1-9. [CrossRef]
- Thatcher A. Web search strategies: The influence of Web experience and task type. Information Processing & Management 2008 May;44(3):1308-1329. [CrossRef]
- Tu Y, Shih M, Tsai C. Eighth graders’ web searching strategies and outcomes: The role of task types, web experiences and epistemological beliefs. Computers & Education 2008 Nov;51(3):1142-1153. [CrossRef]
- Çevik YD. Predicting college students’ online information searching strategies based on epistemological, motivational, decision-related, and demographic variables. Computers & Education 2015 Dec;90:54-63. [CrossRef]
- Willoughby T, Anderson SA, Wood E, Mueller J, Ross C. Fast searching for information on the Internet to use in a learning context: The impact of domain knowledge. Computers & Education 2009 Apr;52(3):640-648. [CrossRef]
- Tsai M, Liang J, Hou H, Tsai C. University students' online information searching strategies in different search contexts. AJET 2012 Jul 26;28(5). [CrossRef]
- Armstrong CJ, Large JA. Manual of Online Search Strategies. London: Routledge; Dec 20, 2018.
- Prensky M. Digital Natives, Digital Immigrants Part 1. On the Horizon 2001 Sep;9(5):1-6. [CrossRef]
- Medisuch. 2017. Zertifizierungsverfahren Medisuch URL: http://www.medisuch.de/zertifizierungsverfahren.asp [accessed 2019-01-29]
- LaValley SA, Kiviniemi MT, Gage-Bouchard EA. Where people look for online health information. Health Info Libr J 2017 Jun 21;34(2):146-155. [CrossRef] [Medline]
- Pauer F, Göbel J, Storf H, Litzkendorf S, Babac A, Frank M, et al. Adopting Quality Criteria for Websites Providing Medical Information About Rare Diseases. Interact J Med Res 2016 Aug 25;5(3):e24 [FREE Full text] [CrossRef] [Medline]
- Google. URL: https://www.google.de/ [accessed 2019-01-29]
- DocCheck. URL: https://www.doccheck.com/welcome [accessed 2019-01-29]
- Morae. Okemos, MI: TechSmith Corporation; 2011. URL: https://www.techsmith.com/tutorial-morae-current.html
- Currie L, Devlin F, Emde J, Graves K. Undergraduate search strategies and evaluation criteria. New Library World 2010 Mar 30;111(3/4):113-124. [CrossRef]
- Holman L. Millennial Students' Mental Models of Search: Implications for Academic Librarians and Database Developers. The Journal of Academic Librarianship 2011 Jan;37(1):19-27. [CrossRef]
- Bloom B, Deyrup MM. The SHU research logs: student online search behaviors trans-scripted. The Journal of Academic Librarianship 2015 Sep;41(5):593-601. [CrossRef]
- van Deursen AJAM. Internet skill-related problems in accessing online health information. Int J Med Inform 2012 Jan;81(1):61-72. [CrossRef] [Medline]
- O'Doherty D, Lougheed J, Hannigan A, Last J, Dromey M, O'Tuathaigh C, et al. Internet skills of medical faculty and students: is there a difference? BMC Med Educ 2019 Jan 30;19(1):39 [FREE Full text] [CrossRef] [Medline]
- Wang L, Wang J, Wang M, Liang J, Xu D. User experience evaluation of Google search for obtaining medical knowledge: a case study. IJDMB 2011;5(6):626-639. [CrossRef]
- Mesko B, Győrffy Z, Kollár J. Digital literacy in the medical curriculum: a course with social media tools and gamification. JMIR Med Educ 2015 Oct 01;1(2):e6 [FREE Full text] [CrossRef] [Medline]
- Scott K, Morris A, Marais B. Medical student use of digital learning resources. Clin Teach 2018 Feb 16;15(1):29-33. [CrossRef] [Medline]
- Ananiadou K, Claro M. 21st century skills and competences for new millennium learners in OECD countries. OECD Education Working Papers 2009. [CrossRef]
- Tsai M, Tsai C. Information searching strategies in web-based science learning: the role of internet self-efficacy. Innovations in Education and Teaching International 2003 Jan;40(1):43-50. [CrossRef]
- Guo RX, Dobson T, Petrina S. Digital natives, digital immigrants: an analysis of age and ict competency in teacher education. Journal of Educational Computing Research 2008 Jun 02;38(3):235-254. [CrossRef]
- Thornham H, McFarlane A. Discourses of the digital native. Information, Communication & Society 2011 Mar;14(2):258-279. [CrossRef]
- Zimmermann M, Jucks R. How experts' use of medical technical jargon in different types of online health forums affects perceived information credibility: randomized experiment with laypersons. J Med Internet Res 2018 Jan 23;20(1):e30 [FREE Full text] [CrossRef] [Medline]
- van Deursen AJAM, van Dijk JAGM. Measuring internet skills. International Journal of Human-Computer Interaction 2010 Sep 17;26(10):891-916. [CrossRef]
|ANOVA: analysis of variance|
|HONcode: Health On the Net Foundation Code of Conduct|
|IT: information technology.|
Edited by C Lovis; submitted 16.09.19; peer-reviewed by I Montagni, C Luz, M Dahl, P Simon; comments to author 26.10.19; revised version received 04.12.19; accepted 10.01.20; published 02.03.20Copyright
©Teresa Loda, Rebecca Erschens, Florian Junne, Andreas Stengel, Stephan Zipfel, Anne Herrmann-Werner. Originally published in JMIR Medical Informatics (http://medinform.jmir.org), 02.03.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as this copyright and license information must be included.