JMI JMIR Med Inform JMIR Medical Informatics 2291-9694 JMIR Publications Toronto, Canada v10i6e34204 35713954 10.2196/34204 Viewpoint Viewpoint Quality Criteria for Real-world Data in Pharmaceutical Research and Health Care Decision-making: Austrian Expert Consensus Lovis Christian Van Biesen Wim Wu Xia Wickramasekera Nyantara Klimek Peter PhD 1 2 https://orcid.org/0000-0003-1187-6713 Baltic Dejan MD 3 https://orcid.org/0000-0001-9855-8464 Brunner Martin MD 4 https://orcid.org/0000-0002-6922-7559 Degelsegger-Marquez Alexander PhD 5 https://orcid.org/0000-0002-0493-4711 Garhöfer Gerhard MD 4 https://orcid.org/0000-0001-5547-5669 Gouya-Lechner Ghazaleh MD 3 https://orcid.org/0000-0002-6320-8015 Herzog Arnold DI 6 https://orcid.org/0000-0003-3845-7674 Jilma Bernd MD 4 https://orcid.org/0000-0001-5652-7977 Kähler Stefan PhD 7 https://orcid.org/0000-0002-5142-1265 Mikl Veronika MA 3 https://orcid.org/0000-0002-3168-691X Mraz Bernhard MA 3 https://orcid.org/0000-0002-9166-3605 Ostermann Herwig PhD 5 https://orcid.org/0000-0003-1412-658X Röhl Claas Ing 8 https://orcid.org/0000-0003-0320-9435 Scharinger Robert MA 9 https://orcid.org/0000-0002-2907-3575 Stamm Tanja PhD 4 https://orcid.org/0000-0003-3073-7284 Strassnig Michael PhD 10 https://orcid.org/0000-0003-1395-0868 Wirthumer-Hoche Christa PhD 6 https://orcid.org/0000-0002-9583-4985 Pleiner-Duxneuner Johannes MD 3
Gesellschaft für Pharmazeutische Medizin Engelhorngasse 3 Vienna, 1210 Austria 43 1 40160 ext 36255 johannes.pleiner-duxneuner@roche.com
https://orcid.org/0000-0002-3492-846X Institute for Science of Complex Systems Center for Medical Statistics, Informatics, and Intelligent Systems Medical University of Vienna Vienna Austria Complexity Science Hub Vienna Vienna Austria Gesellschaft für Pharmazeutische Medizin Vienna Austria Medical University of Vienna Vienna Austria Gesundheit Österreich GmbH Vienna Austria Austrian Medicines and Medical Devices Agency (AGES Medizinmarktaufsicht) Vienna Austria Verband der pharmazeutischen Industrie Österreichs (PHARMIG) Vienna Austria EUPATI Austria Vienna Austria Federal Ministry of Social Affairs, Health, Care and Consumer Protection Vienna Austria Vienna Science and Technology Fund Vienna Austria Corresponding Author: Johannes Pleiner-Duxneuner johannes.pleiner-duxneuner@roche.com 6 2022 17 6 2022 10 6 e34204 11 10 2021 27 4 2022 16 5 2022 17 5 2022 ©Peter Klimek, Dejan Baltic, Martin Brunner, Alexander Degelsegger-Marquez, Gerhard Garhöfer, Ghazaleh Gouya-Lechner, Arnold Herzog, Bernd Jilma, Stefan Kähler, Veronika Mikl, Bernhard Mraz, Herwig Ostermann, Claas Röhl, Robert Scharinger, Tanja Stamm, Michael Strassnig, Christa Wirthumer-Hoche, Johannes Pleiner-Duxneuner. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 17.06.2022. 2022

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.

Real-world data (RWD) collected in routine health care processes and transformed to real-world evidence have become increasingly interesting within the research and medical communities to enhance medical research and support regulatory decision-making. Despite numerous European initiatives, there is still no cross-border consensus or guideline determining which qualities RWD must meet in order to be acceptable for decision-making within regulatory or routine clinical decision support. In the absence of guidelines defining the quality standards for RWD, an overview and first recommendations for quality criteria for RWD in pharmaceutical research and health care decision-making is needed in Austria. An Austrian multistakeholder expert group led by Gesellschaft für Pharmazeutische Medizin (Austrian Society for Pharmaceutical Medicine) met regularly; reviewed and discussed guidelines, frameworks, use cases, or viewpoints; and agreed unanimously on a set of quality criteria for RWD. This consensus statement was derived from the quality criteria for RWD to be used more effectively for medical research purposes beyond the registry-based studies discussed in the European Medicines Agency guideline for registry-based studies. This paper summarizes the recommendations for the quality criteria of RWD, which represents a minimum set of requirements. In order to future-proof registry-based studies, RWD should follow high-quality standards and be subjected to the quality assurance measures needed to underpin data quality. Furthermore, specific RWD quality aspects for individual use cases (eg, medical or pharmacoeconomic research), market authorization processes, or postmarket authorization phases have yet to be elaborated.

 real-world data real-world evidence data quality data quality criteria RWD quality recommendations pharmaceutical research health care decision-making quality criteria for RWD in health care Gesellschaft für Pharmazeutische Medizin GPMed Introduction

Real-world data (RWD) is an overarching term for data on patient’s health (health status, effectiveness, medical treatment, the pattern of use of medicinal products, and resource use, etc) that are collected in routine health care processes and not in the context of clinical trials. RWD involve large and complex data sets such as data from electronic health records, pharmacy data, electronic smart devices, patient-reported outcomes, and digital applications or platforms [1,2]. When RWD are analyzed, they lead to real-world evidence (RWE) on the pattern of use and effectiveness of any kind of procedure, drug, or nonpharmacological intervention. The availability of RWD and evolving analytic techniques to generate RWE have created interest within the research and medical communities to use RWD and RWE to enhance clinical research and support regulatory decision-making [1,3]. On a European level, the European Medicines Agency (EMA) and Heads of Medicines Agencies fully recognize the value of health data and set up a joint task force to describe the health data landscape from a regulatory perspective and identify practical steps for the European medicines regulatory network to make the best use of health data in support of innovation and public health in the European Union [4].

The comprehensive work plan identifies 10 priorities [5], such as delivering a sustainable platform to access and analyze health care data from across the European Union (Data Analysis and Real World Interrogation Network [6]) or establishing an EU framework for data quality (European Health Data & Evidence Network [7] and Health Outcomes Observatory [8]) and representativeness. Despite many initiatives, there are still no guidelines for the quality criteria that RWD must meet in order to be able to use it for decision-making purposes within regulatory or routine clinical decision support. As a first example, the EMA Guideline on registry-based studies [9] provides considerations on good practice for registries to increase their usefulness for regulatory purposes.

The objective of this consensus statement of the Austrian Expert Group led by Gesellschaft für Pharmazeutische Medizin (GPMed; Austrian Society for Pharmaceutical Medicine) is to provide an overview and first recommendations for the quality criteria of RWD for primary and secondary research purposes to be adopted in medical or pharmacoeconomic research and health care decision-making processes. The consensus statement does not discuss the general use of RWD nor how to obtain RWE in general.

 Methods

After EMA published a drafted guideline for registry-based studies, interested GPMed board members volunteered together with Austrian Medicines and Medical Devices Agency executive experts to assess how ready the Austrian research landscape is for registry-based studies.

The Austrian Medicines and Medical Devices Agency and GPMed invited Austrian RWD researchers and data experts to contribute voluntarily to the topic. The criteria to select working group members were those with scientific work in the field and longstanding expertise in using RWD for research purposes. After the kickoff meeting in April 2021, the expert group led by GPMed met on a monthly basis; reviewed guidelines, frameworks, use cases, or viewpoints; and derived a consensus statement on the quality criteria for RWD to be used more effectively for medical research purposes beyond the registry-based studies discussed in the EMA Guideline for registry-based studies [9].

Following agreement on a joint definition on RWD, experts from the group shared examples of RWD frameworks, guidelines, or viewpoints, which were discussed in the working group, and consensus was reached unanimously within the monthly meetings.

 Results Definition of RWD

Despite an increasing recognition of the value of RWD, a global consensus on the definition of RWD is lacking [10]. The definition of RWD can differ in various areas of application (eg, public health vs automotive industry). However, the expert group led by GPMed reviewed several definitions [7,8,10-15] and agreed on the following description.

Real-world data can be defined as data relating to patient health status or the delivery of health care that are routinely collected from a variety of sources (including patient-reported outcomes), such as:

health care databases (systems into which health care providers routinely enter clinical and laboratory data; eg, electronic health records and pharmacist databases),

health insurance and claims databases (maintained by payers for reimbursement purposes),

patient registries (data on a group of patients with specific characteristics in common),

disease registries (data on a particular disease or disease-related patient characteristic regardless of exposure to any medicinal product, other treatment, or a particular health service),

data gathered from other sources that can inform on health status, such as mobile devices, wearables, or other smart medicinal products (eg, real-time continuous glucose monitoring devices),

social media– and patient-powered research networks (eg, patient networks to share health information),

biobanks, and

observational studies.

Note that this definition includes data that are neither collected by licensed medical devices operated by health professionals in clinical settings nor observational data that are typically stored in public health registries and administrative databases. Namely, RWD also include health-related data that are generated by the patient by means of digital health technologies (sensors, wearables, and smartphones, etc). Hence, ethical and regulatory frameworks should also be applied to these health-related data and not only target health care databases and registries [16].

 Examples of RWD Frameworks

Globally and Europe-wide, more and more examples of how RWD are used for research or regulatory purposes are being published. The expert group decided to illustrate some examples of how the quality of RWD is ensured along different approaches (Table 1). Further details to this overview can be found in the Multimedia Appendix 1.

Examples and short descriptions of reviewed real-world data (RWD) frameworks.

RWD framework Short description Country
RWD for health systems research [17-23] Nordic countries have set the worldwide gold standard for how RWD can be leveraged. Good RWD frameworks exist in Finland, Denmark, Sweden, Iceland, and Norway. The RWD quality and infrastructure built up in these countries can be seen as best practice examples for how to leverage RWD for research. Denmark, Finland, Iceland, Norway, and Sweden
Danish Data Analytics Center [24] The Danish DACa has access to some of the most sophisticated and complete patient-level health data in the world and meets the highest requirements for data and IT security. DAC constitutes a unique possibility for the use of big data analytics to discover hidden patterns to benefit patients. It will reduce the entry barriers for new drugs to go to market while maintaining the high safety standards currently in place. Denmark
EMAb submission supported by historical cohort patient data [25] Based on the observed efficacy in Phase 2 studies (n=189 and n=36) and combined with an additional historical comparator study (1139 cases), conditional marketing authorization was granted with the need to better quantify the magnitude of the effect by submitting data from a Post Authorization Efficacy Study (Phase 3 randomized, comparative study of blinatumomab vs standard of care chemotherapy) as well as a noninterventional Post Authorization Safety Study in subsequent years. European Union
Demonstrated the research potential of a clinico-genomic database [26,27] In 2017, Foundation Medicine and Flatiron Health created a proof-of-concept study. Using a sample size of over 2000 patients with non–small cell lung cancer, they discovered that high versus low tumor mutation burden showed a far stronger association than high versus low PD-L1 levels after immunotherapy. Their results were nearly identical to those derived by a drug manufacturer from a post hoc analysis of a failed clinical trial. The validation study helped establish the groundwork for this data set to be used to advance cancer research. United States
Multidatabase studies for medicines surveillance in real-world settings [28,29] Postmarketing studies can be underpowered if outcomes or exposure of interest are rare, or the interest is in the subgroup effects. Combining several databases might provide the statistical power needed. Although many multidatabase studies have been performed in Europe in the past 10 years, there is a lack of clarity on the peculiarities and implications of the existing strategies to conduct them. Experts identified 4 strategies to execute multidatabase studies, classified according to specific choices in the execution. European Union
EUnetHTAc REQueSTd [30] The Registry Evaluation and Quality Standards Tool (REQueST) aims to support health technology assessment organizations and other actors in guiding and evaluating registries for effective use in health technology assessment. European Union

aDAC: Data Analytics Center.

bEMA: European Medicines Agency.

cEUnetHTA: European Network for Health Technology Assessment.

dREQueST: Registry Evaluation and Quality Standards Tool.

 Legal Frameworks

The current legal framework in Austria with the Federal Statistics Act as well as the Research Organization Act recognizes the “use” of RWD—especially for research purposes [31-33].

Independently of the question of data availability, many RWD sources, as defined within this expert consensus paper, do not address data quality issues. Therefore, the need for high–data quality standards should be also recognized by legal frameworks. On a European level, data quality aspects are strongly embedded within the development of the European Health Data Space [34] and Data Analysis and Real World Interrogation Network [6]. Shared outcomes on data quality should be reflected within local legal frameworks as well.

 Recommendations Data Quality

RWD are often used for purposes that are different from the intention for which the data were collected originally. Therefore, it is of utmost importance to check upfront if the RWD are adequate in terms of clearly defined quality criteria and can, therefore, be used in general for primary or secondary research purposes as well. Due to the lack of guidelines defining the quality standards of RWD to be used for decision-making, it is even more important to be able to assess the suitability of RWD for research purposes by applying checklists and some standardized questionnaires [35-38].

 RWD Should Follow High Standards and Be Subject to Quality Assurance

The value of the secondary use of RWD data (in particular, registries) for research purposes depends crucially on their quality as quantified by completeness and accuracy [39], next to timeliness, comparability, the technical prerequisite that the size of the data source is sufficient (ie, the study does not become underpowered), and that the data is in principle accessible and can be mapped with other relevant data sets (well defined research question outlined in a research plan). An evaluation with regard to these factors is therefore recommended before using the data. Note that these quality criteria are not unique in the sense that alternative data quality concepts have also been described (eg, validity, consistency, and integrity).

Completeness is defined as the proportion of true cases of a variable (disease, treatment, and diagnose, etc) in all or a certain subgroup of patients that is correctly reported in the data. Completeness therefore captures the amount of missing data in a specific source—the extent to which all necessary data that could have been registered has been registered [40]. Very often there is no comprehensive reference source available for evaluating the completeness of a data set with regard to the general population. In that case, it might be advisable to identify studies that report the variables of interest for specific comparable subgroups and therefore allow for an assessment of data completeness [39]. These comparisons should ideally be performed on an individual level (eg, comparing data records from registries for certain diseases to administrative records) or, in cases where the required information is not available on an individual level, attempts should be made to examine completeness at least on an aggregate level (by comparing the expected number of cases across data sets).

Accuracy measures the proportion of patients with a certain property (diagnosis, prescription, and socioeconomic or demographic properties, etc) in a data set that truly have the property. Accuracy is typically assessed by comparing the data records with the reference standard used to confirm the specific variable [41]. In many cases, this reference could be the medical record; for certain areas, other references might be feasible as well. One strategy to perform such a comparison could be to randomly sample a given percentage (eg, 5%) or an absolute number (eg, 1000) manually. This helps to identify errors and whether they are systematic (as often happens through algorithmic problems when the data are collected in an automated way or if the data are collated from different reporting systems, regional or otherwise) or random (often resulting from manual data collection), thereby informing strategies to increase data accuracy.

Timeliness measures data quality with regard to the time at which the variable (disease and diagnosis, etc) was recorded (eg, the extent to which the time of the recorded disease corresponds to the true time of the disease). This can often be assessed together with completeness and accuracy and is of particular importance in longitudinal study designs.

Furthermore, comparability needs to be checked to ensure that variable definitions in a data set conform to international guidelines and other relevant references.

A comprehensive review of 114 data quality studies in the Danish registry network showed that both completeness and accuracy increased over time and accuracy varies substantially across different diseases, between less than 15% of correctly coded diagnoses to almost 100% [41]. This finding underscores the need for data quality assurance of RWD for research use.

 High Research Standards Should Underpin the Quality of RWD Study Protocol

Observational postmarketing studies are an important tool, using data obtained from routine clinical care, to provide data on medical treatment effect estimates and the tolerability of medicinal products in a real-world setting, as well as for medical devices as part of the postmarketing surveillance [42]. Nonrandomized studies may be used to complement the evidence base represented by randomized controlled trials [43], even though one cannot expect nonrandomized, observational studies to exactly reproduce randomized controlled trials as these are different study designs, and hence measure different types of effects [44]. Noncontrolled studies lack a comparison group, which means that inferences on the treatment effect and tolerability must rely on before-and-after comparisons of the outcome of interest. Treatment effect estimates and tolerability derived from nonrandomized studies are at greater risk of bias. Thus, data from routine clinical observation should be collected after the development of a study protocol where the population of interest, study outcome, methods for data generation and analysis, limitation of study data, and bias are defined in advance, as also defined in the EMA guideline for registry-based studies [9].

 Informed Consent

The informed consent process of patients in observational, noninterventional studies are not discussed by Good Clinical Practice (ISO 14155) [45], and this topic is still dealt with heterogeneously throughout the European Union. Within the study protocol, the consent process and requirements of compliance to the General Data Protection Regulation (GDPR) should be specified. Data generated in an anonymized way would not require patient consent, though collection of pseudonymized data in observational studies requires the consent of patients prior to data collection, which should be limited only to the GDPR requirements, and not include any consent to medical treatment. The burden of obtaining informed consent to collect routine clinical data should be kept feasible to reduce bias of missing data from severely ill patients or patients incapable of consenting, such as in emergency situations. Since GDPR applies only to living people, a waiver for data collection from the deceased can be obtained if the purpose is sufficiently outlined in the study protocol.

 Institutional Review Board and Ethics Committee

Within the study protocol, all interventions in the observational trial (ie, treatment, diagnostic or monitoring procedures) should fall within the standard of care or routine treatment, as interpreted by the competent authority or ethics committee in that member state. Thus, a review and approval from the respective ethics committee is required, as also indicated in the EMA guideline for registry-based studies [9].

 Checklist on Quality Criteria for RWD

Following general recommendations and reflecting guidelines and checklists on registry-based research [9,37], the expert group suggests a minimum set of criteria summarized within the checklist presented in Table 2 to ensure the quality of RWD for research purposes and health care decision-making processes.

Gesellschaft für Pharmazeutische Medizin (GPMed) checklist for real-world data (RWD) quality.

Criteria Description
Data management and stewardship

“FAIR Data Principles” which formulate principles that sustainable, reusable research data and research data infrastructures must meet [38,46,47]
Governance framework

Available policy for collaborations with external organizations

Involvement of patient organizations

Governance structure for decision-making on requests for collaboration

Templates for research and data-sharing contracts between partners and institutions
Quality requirements

High–RWD quality standards are implemented, such as completeness, accuracy, timeliness, and comparability

Process in place for ongoing data quality assessments

Processes in place for quality planning, control, assurance, and improvement

Data verification (the method and frequency of verification)

Auditing practice
Data privacy and transparency

Informed consent processes and its validity for research purposes according to General Data Protection Regulation and relevant national regulations

Data privacy officer
Research objectives

Well-defined research question outlined in a research plan

Available documentation, protocol, or proposal that describes the purpose of RWD use and rational that the RWD sources adequately address the research questions (eg, study protocol)

Approval of RWD use from independent an institutional review board or ethics committee

Protocol should follow the Declaration of Helsinki, and furthermore, the Declaration of Taipei [48] on Research on Health Databases, Big Data and Biobanks should be taken into account
Data providers

Adequate description of data providers, such as patients, caregivers, or health care professionals; their geographical area; and any selection process (inclusion and exclusion criteria) that may be applied for their acceptance as data providers
Patient population covered

Adequate description of the type of patient population (disease, condition, time period covered, and procedure), which defines the criteria for patient eligibility

Relevance of setting and catchment area

Clarity on patients’ inclusion and exclusion criteria

Methods applied to minimize selection bias and loss to follow-up

Ensure fair representations of minorities, sex, gender, and socially disadvantaged groups
Data elements

Core RWD set collected for RWD use case or purpose

Definition, dictionary, and format of data elements

Standards and terminologies applied

Capabilities and plans for amendments of data elements
Infrastructure

High-quality systems for RWD collection, recording, and reporting, including timelines

Capability (and experience) for expedited reporting and evaluation of severe suspected adverse reactions in RWD collection

Capability (and experience) for periodic reporting of clinical outcomes—ideally patient-reported outcomes—and adverse events reported by physicians, at the individual-patient level and aggregated data level

Capability (and experience) for data cleaning, extraction, transformation, and analysis

Capability (and experience) for data transfer to external organizations

Capabilities for amendment of safety reporting processes
Discussion Principle Findings

Over the past months, EU and EMA strategies, workplans, and initiatives on health data use developed very quickly [34,49-51]. This paper shows the consensus of a multistakeholder expert group which summarizes a minimum set of the quality criteria of RWD for research and decision-making purposes in health care. The most important quality assurance measures identified are a profound data management and stewardship; established governance framework; standardized quality requirements; adhered data privacy and transparency measures; well-defined research objectives; adequate description of data providers; well-described patient population covered; outlined which data elements are required; and high-quality infrastructure for RWD collection, recording, and reporting.

 Conclusions

To future-proof registry-based studies, the group strongly recommends that RWD should follow high standards and be subject to the quality assurance measures needed to underpin the quality of RWD. Furthermore, specific RWD quality aspects for individual use cases (eg, medical or pharmacoeconomic research), market authorization processes, or postmarket authorization phases have yet to be elaborated.

Examples of real-world data frameworks or use cases.

 Abbreviations EMA

European Medicines Agency

 GDPR

General Data Protection Regulation

 GPMed

Gesellschaft für Pharmazeutische Medizin

 RWD

real-world data

 RWE

real-world evidence

The work was supported in kind by the participating organizations. The authors declare no financial support or funding for this project.

DB is an employee of Amgen GmbH, Vienna, Austria. SK is an employee of Bristol Myers Squibb, Vienna, Austria. BM is an employee of Novartis Pharma GmbH, Austria. TS reports grants and personal fees from AbbVie, Roche, Sanofi, Takeda, and Novartis, all outside the submitted work. VM and JPD are employees of Roche Austria GmbH. All other authors declare no other conflicts of interest.

 Submitting documents using real-world data and real-world evidence to FDA for drugs and biologics: guidance for industry U.S. Food & Drug Administration 2019 05 2021-09-17 https://www.fda.gov/media/124795/download Sherman RE Anderson SA Dal Pan Gerald J Gray GW Gross T Hunter NL LaVange L Marinac-Dabic D Marks PW Robb MA Shuren J Temple R Woodcock J Yue LQ Califf RM Real-world evidence - what is it and what can it tell us? N Engl J Med 2016 12 08 375 23 2293 2297 10.1056/NEJMsb1609216 27959688 Babrak LM Smakaj E Agac T Asprion PM Grimberg F der Werf DV van Ginkel EW Tosoni DD Clay I Degen M Brodbeck D Natali E Schkommodau E Miho E RWD-Cockpit: application for quality assessment of real-world data JMIR Form Res. Preprint posted online on Feb 19, 2022. 10.2196/29920 35266872 Big Data Steering Group workplan 2021-2023 European Medicines Agency 2021-09-17 https://www.ema.europa.eu/en/documents/work-programme/workplan-2021-2023-hma/ema-joint-big-data-steering-group_en.pdf Priority recommendations of the HMA-EMA joint Big Data Task Force European Medicines Agency 2021-09-17 https://www.ema.europa.eu/en/documents/other/priority-recommendations-hma-ema-joint-big-data-task-force_en.pdf Data Analysis and Real World Interrogation Network (DARWIN EU) European Medicines Agency 2021-09-17 https://www.ema.europa.eu/en/about-us/how-we-work/big-data/data-analysis-real-world-interrogation-network-darwin-eu Result of the 4th open call for data partners European Health Data & Evidence Network 2021-09-17 https://us20.campaign-archive.com/?u=123c73def0355ab534c08baa9&id=214d8c4a9e Stamm T Bott N Thwaites R Mosor Erika Andrews MR Borgdorff J Cossio-Gil Y de Portu S Ferrante M Fischer F Hameed F Hazelzet J Hopkins D Kalra D Metcalfe T Molero E Newson R Patalano F Prasser F Rose M Lindskov Sachs M Soderberg J Strammiello V van de Poll L Styliadou M Building a value-based care infrastructure in Europe: the Health Outcomes Observatory NEJM Catalyst 2021 06 09 10.1056/CAT.21.0146 Committee for Human Medicinal Products (CHMP) Guideline on registry-based studies European Medicines Agency 2021 10 22 2022-05-05 https://www.ema.europa.eu/en/documents/scientific-guideline/guideline-registry-based-studies_en-0.pdf Makady A de Boer Anthonius Hillege H Klungel Olaf Goettsch Wim What is real-world data? a review of definitions based on literature and stakeholder interviews Value Health 2017 07 01 20 7 858 865 10.1016/j.jval.2017.03.008 28712614 S1098-3015(17)30171-7 HMA-EMA Joint Big Data Taskforce phase II report: 'evolving data-driven regulation' European Medicines Agency 2019 2021-05-11 https://www.ema.europa.eu/en/documents/other/hma-ema-joint-big-data-taskforce-phase-ii-report-evolving-data-driven-regulation_en.pdf Read-world evidence U.S. Food & Drug Administration 2021-05-11 https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence Corrigan-Curay J Sacks L Woodcock J Real-world evidence and real-world data for evaluating drug safety and effectiveness JAMA 2018 09 04 320 9 867 868 10.1001/jama.2018.10136 30105359 2697359 MHRA position statement and guidance: electronic health records Medicines & Healthcare Products Regulatory Agency 2015 09 2022-05-05 https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/470228/Electronic_Health_Records_MHRA_Position_Statement.pdf Here, no one goes at it alone PatientsLikeMe 2021-09-21 https://www.patientslikeme.com/ Vayena E Haeusermann T Adjekum A Blasimme A Digital health: meeting the ethical and policy challenges Swiss Med Wkly 2018 148 w14571 10.4414/smw.2018.14571 29376547 smw-14571 Sørensen H T Regional administrative health registries as a resource in clinical epidemiology: a study of options, strengths, limitations and data quality provided with examples of use Int J Risk Saf Med 1997 10 1 1 22 10.3233/JRS-1997-10101 23511270 UK7072674664272X Sund R Quality of the Finnish Hospital Discharge Register: a systematic review Scand J Public Health 2012 08 16 40 6 505 15 10.1177/1403494812456637 22899561 1403494812456637 Lynge E Sandegaard JL Rebolj M The Danish National Patient Register Scand J Public Health 2011 07 20 39 7 Suppl 30 3 10.1177/1403494811401482 21775347 39/7_suppl/30 Ludvigsson JF Andersson E Ekbom A Feychting M Kim J Reuterwall C Heurgren M Olausson PO External review and validation of the Swedish national inpatient register BMC Public Health 2011 06 09 11 1 450 10.1186/1471-2458-11-450 21658213 1471-2458-11-450 PMC3142234 Gudbjornsson B Thorsteinsson SB Sigvaldason H Einarsdottir R Johannsson M Zoega H Halldorsson M Thorgeirsson G Rofecoxib, but not celecoxib, increases the risk of thromboembolic cardiovascular events in young adults-a nationwide registry-based study Eur J Clin Pharmacol 2010 06 16 66 6 619 25 10.1007/s00228-010-0789-2 20157701 Norsk pasientregister (NPR) Norwegian Directorate of Health 2022-06-09 https://www.helsedirektoratet.no/tema/statistikk-registre-og-rapporter/helsedata-og-helseregistre/norsk-pasientregister-npr Schmidt M Schmidt SAJ Sandegaard JL Ehrenstein V Pedersen L Sørensen Henrik Toft The Danish National Patient Registry: a review of content, data quality, and research potential Clin Epidemiol 2015 11 17 7 449 90 10.2147/CLEP.S91125 26604824 clep-7-449 PMC4655913 Data Analytics Centre Danish Medicines Agency 2021-09-21 https://laegemiddelstyrelsen.dk/en/about/organisation/name/ Assessment report: BLINCYTO, international non-proprietary name: blinatumomab European Medicines Agency 2022-05-30 https://www.ema.europa.eu/en/documents/assessment-report/blincyto-epar-public-assessment-report_en.pdf Getting closer to cancer research's holy grail: the Clinico-Genomic Database Roche 2019 04 16 2021-05-21 https://www.roche.com/about/priorities/personalised_healthcare/combining-data-to-advance-personalised-healthcare.htm Singal G Miller PG Agarwala V Li G Kaushik G Backenroth D Gossai A Frampton GM Torres AZ Lehnert EM Bourque D O'Connell C Bowser B Caron T Baydur E Seidl-Rathkopf K Ivanov I Alpha-Cobb G Guria A He J Frank S Nunnally AC Bailey M Jaskiw A Feuchtbaum D Nussbaum N Abernethy AP Miller VA Association of patient characteristics and tumor genomics with clinical outcomes among patients with non-small cell lung cancer using a clinicogenomic database JAMA 2019 04 09 321 14 1391 1399 10.1001/jama.2019.3241 30964529 2730114 PMC6459115 Gini R Sturkenboom MCJ Sultana J Cave A Landi A Pacurariu A Roberto G Schink T Candore G Slattery J Trifirò Gianluca Working Group 3 of ENCePP (Inventory of EU data sourcesmethodological approaches for multisource studies) Different strategies to execute multi-database studies for medicines surveillance in real-world setting: a reflection on the European model Clin Pharmacol Ther 2020 08 108 2 228 235 10.1002/cpt.1833 32243569 PMC7484985 Real world research on medicines: contribution of the European Network of Centres in Pharmacoepidemiology and Pharmacovigilance (ENCePP) European Medicines Agency 2021 03 08 2021-05-21 https://www.ema.europa.eu/en/events/real-world-research-medicines-contribution-european-network-centres-pharmacoepidemiology#event-summary-section REQueST Tool and its vision paper EUnetHTA 2021-05-21 https://eunethta.eu/request-tool-and-its-vision-paper/ Registerforschung Austrian Federal Ministry of Education, Science and Research 2021-09-21 https://www.bmbwf.gv.at/Themen/Forschung/Forschung-in-%C3%96sterreich/Strategische-Ausrichtung-und-beratende-Gremien/Leitthemen/Registerforschung.html König T Schmoigl L Erfolgreiche Registerforschung in Österreich: Welchen Mehrwert generiert die reglementierte Öffnung von Registerdaten für die wissenschaftliche Forschung? Eine Darstellung anhand von drei Beispielen Österreichisches Institut für Wirtschaftsforschung (WIFO) und Institut für höhere Studien (IHS) 2020 11 2021-09-21 https://irihs.ihs.ac.at/id/eprint/5576/7/koenig-schmoigl-2020-erfolgreiche-registerforschung-in-oesterreich.pdf König T Strassnig M Schwarz G Oberhofer H Zugang zu Register- und Individualdaten für die wissenschaftliche Forschung in Österreich fteval Journal for Research and Technology Policy Evaluation 2020 50 11 15 10.22163/fteval.2020.464 European Health Data Space European Commission 2022-05-04 https://ec.europa.eu/health/ehealth/dataspace_en A validated checklist for evaluating the quality of observational cohort studies for decision-making support Grace Principles 2021-07-12 https://www.graceprinciples.org/doc/GRACE-Checklist-031114-v5.pdf Motheral B Brooks J Clark MA Crown WH Davey P Hutchins D Martin BC Stang P A checklist for retrospective database studies--report of the ISPOR Task Force on Retrospective Databases Value Health 2003 03 6 2 90 7 10.1046/j.1524-4733.2003.00242.x 12641858 S1098-3015(10)60137-4 Determining real-world data's fitness for use and the role of reliability Duke-Margolis Center for Health Policy 2019 09 26 2021-09-21 https://healthpolicy.duke.edu/sites/default/files/2019-11/rwd_reliability.pdf Wilkinson MD Dumontier M Aalbersberg IJ Appleton G Axton M Baak A Blomberg N Boiten J da Silva Santos LB Bourne PE Bouwman J Brookes AJ Clark T Crosas M Dillo I Dumon O Edmunds S Evelo CT Finkers R Gonzalez-Beltran A Gray AJ Groth P Goble C Grethe JS Heringa J 't Hoen Peter A C Hooft R Kuhn T Kok R Kok J Lusher SJ Martone ME Mons A Packer AL Persson B Rocca-Serra P Roos M van Schaik R Sansone S Schultes E Sengstag T Slater T Strawn G Swertz MA Thompson M van der Lei J van Mulligen E Velterop J Waagmeester A Wittenburg P Wolstencroft K Zhao J Mons B The FAIR Guiding Principles for scientific data management and stewardship Sci Data 2016 03 15 3 1 160018 10.1038/sdata.2016.18 26978244 sdata201618 PMC4792175 Sorensen H T Sabroe S Olsen J A framework for evaluation of secondary data sources for epidemiological research Int J Epidemiol 1996 04 25 2 435 42 10.1093/ije/25.2.435 9119571 Arts DGT De Keizer NF Scheffer G Defining and improving data quality in medical registries: a literature review, case study, and generic framework J Am Med Inform Assoc 2002 9 6 600 11 10.1197/jamia.m1087 12386111 PMC349377 Schmidt M Schmidt SAJ Sandegaard JL Ehrenstein V Pedersen L Sørensen HT The Danish National Patient Registry: a review of content, data quality, and research potential CLEP 2015 11 449 10.2147/clep.s91125 Directive of the European Parliament and of the Council of 6 October 1997 Amending Council Directives 90/387/EEC and 92/44/EEC for the Purpose of Adaptation to a Competitive Environment in Telecommunications (97/51/EC) The European Parliament and the Council of the European Union 1997 10 29 2022-05-30 https://doi.org/10.1163/9789004481466_018 Hernán Miguel A Robins JM Using big data to emulate a target trial when a randomized trial is not available Am J Epidemiol 2016 04 15 183 8 758 64 10.1093/aje/kwv254 26994063 kwv254 PMC4832051 Groenwold RHH Trial emulation and real-world evidence JAMA Netw Open 2021 03 01 4 3 e213845 10.1001/jamanetworkopen.2021.3845 33783521 2777895 ISO 14155:2020 clinical investigation of medical devices for human subjects — good clinical practice International Organization for Standardization 2020 07 2022-05-30 https://www.iso.org/standard/71690.html FAIR Principles Go Fair 2021-09-21 https://www.go-fair.org/fair-principles/ Facile R Muhlbradt EE Gong M Li Q Popat V Pétavy Frank Cornet R Ruan Y Koide D Saito TI Hume S Rockhold F Bao W Dubman S Jauregui Wurst B Use of Clinical Data Interchange Standards Consortium (CDISC) standards for real-world data: expert perspectives from a qualitative Delphi survey JMIR Med Inform 2022 01 27 10 1 e30363 10.2196/30363 35084343 v10i1e30363 PMC8832264 Declaration of Taipei: research on health databases, big data and biobanks World Medical Association 2021-09-21 https://www.wma.net/what-we-do/medical-ethics/declaration-of-taipei/ Joint action towards the European Health Data Space - TEHDAS TEHDAS 2022-05-04 https://tehdas.eu/ Big Data Steering Group Workplan 2021-2023 European Medicines Agency 2022-05-04 https://www.ema.europa.eu/en/documents/work-programme/workplan-2021-2023-hma/ema-joint-big-data-steering-group_en.pdf Electronic cross-border health services European Commission 2022-05-04 https://ec.europa.eu/health/ehealth/electronic_crossborder_healthservices_en