This was a retrospective cohort study. The study population comprised all the inpatient stays from 2007 to 2014 in a set of French, Danish, and Bulgarian hospitals (see Inpatient Stays section) participating in the European “Patient Safety through Intelligent Procedures“ (PSIP) project [37]. A set of DDI rules was defined, including causes (a VKA and another drug) and potential outcomes (VKA potentiation or inhibition, as defined in the Set of DDI Rules section). The causes and the potential outcomes were retrospectively tracked over time in the data set, and the probability of each outcome was estimated automatically for each DDI rule.

We reanalyzed 96,378 inpatient stays in a French community hospital, 53,635 inpatient stays in a group of Danish community hospitals, and 6880 inpatient stays in a Bulgarian hospital. Only stays with at least one laboratory INR result and at least one day with VKA administration were included. Those data had been collected exhaustively during routine patient care. The available data [9] included (1) demographic and administrative information (eg, age, gender, and dates), (2) diagnoses coded according to the International Statistical Classification of Diseases and Related Health Problems, 10th Revision [38], (3) daily drug administrations, encoded using the Anatomical Therapeutic Chemical (ATC) Classification System terminology [39], and (4) laboratory results encoded using the Clinical Nomenclature for Properties and Units terminology [40].

We used the combined results of three literature reviews (Holbrook et al [7], Nutescu et al [5], and Di Minno et al [1]) to identify DDI rules involving VKAs. After deduplication, a list of 149 DDIs (available in Multimedia Appendix 1) was created. We then mapped the drug names to ATC codes [39] by taking into account the active substances and the administration route. The ATC mapping was inclusive and, when appropriate, also involved ATC codes relating to drug combinations.

Of the 149 DDIs, 7 were excluded because they corresponded to drugs without ATC terms. Two drugs had the same ATC code (amoxicillin + tranexamic acid, and amoxicillin + clavulanate) and were therefore combined in 1 DDI. The remaining drugs were variously analgesics, antipyretics, and immunological agents (n=21), anti-infectives (n=47), cardiovascular and anti-hypertensive drugs (n=29), central nervous system drugs (n=19), and other drugs (n=25). Ultimately, we obtained 107 drugs that might potentiate VKAs and 34 drugs that might inhibit VKAs (including 4 drugs that belonged to both categories). A final set of 141 DDI rules was obtained for drug administration. The same number of rules was obtained for drug discontinuation, leading to 2×141 rules in total.

We then obtained DDIs, in the form “VKA & administration of DrugX → outcome” and “VKA & discontinuation of DrugX → reverse outcome,” where the “DrugX” term was a drug that potentially interacted with VKAs, and the “outcome” term was defined as VKA potentiation (INR≥5) or inhibition (INR≤1.5).

In descriptive analyses, qualitative variables were reported as the number and percentage for each category, and quantitative variables were reported as the mean and standard deviation (SD) for symmetric data distributions or the median and interquartile range (IQR) for asymmetric data distributions.

The main objectives of the statistical analysis were to follow up each inpatient stay in which a VKA was administered, detect outcomes over time, and estimate odds ratios (ORs) for the second drug in the DDI rule. The following procedure was applied for each DDI rule. The “VKA & tramadol → INR≥5” rule serves here as an example. Figure 1 shows the data transformation process for a hospital stay with VKA and tramadol (an “exposed stay,” left side) and a stay with VKA but no tramadol (a “nonexposed stay,” right side). The observation periods were designed to reflect each drug’s onset of action and postdiscontinuation duration of action. An “exposed” inpatient started the day after the two drugs had been administered together and ended 4 days after the first of the two was discontinued or after both were discontinued on the same day. A “nonexposed” inpatient started on the day after the VKA had been administered and stopped 4 days after the VKA had been discontinued. The observation period was searched for the outcome (Figure 1).

For each drug, we used the same approach to test whether drug discontinuation would lead to the opposite outcome. For instance, the “VKA & tramadol → INR≥5” rule also enabled us to test the “VKA & tramadol discontinuation → INR≤1.5” rule.

We first computed the unadjusted OR (95% confidence limits [CLs]) for the exposure and the outcome, using the Fisher exact test [41]. We then performed a multivariable logistic regression to predict the outcome. The covariates were the studied drug, age, albuminemia, pre-albuminemia, creatininemia, aspartate transaminase/alanine transaminase (ASAT/ALAT) levels, thyroid stimulating hormone (TSH) level, and N-terminal-pro brain natriuretic peptide (Nt-proBNP) (the last five of these covariates are surrogate markers for malnutrition, kidney failure, liver failure, dysthyroidism, and heart failure, respectively). We thus obtained the adjusted OR (95% CLs). Lastly, the model’s covariates were selected in a stepwise procedure, yielding the “stepwise OR” (95% CLs) [42].

Quantitative variables were placed in classes when the effect was not linear (“ref” denotes the reference class): Age was classified as “<70” (ref), “70-79,” and “≥80”. The albuminemia was classified in g/L as “<30” and “≥30” (ref). Pre-albuminemia was classified in g/L as “<0.07,” “0.07-0.10,” and “≥0.11” (ref). Creatininemia was classified in mg/L as “≤15” (ref), “16-24,” and “≥25”. ASAT/ALAT levels were classified in IU/L as “<250” (ref) and “≥250”. TSH levels were classified in mU/L as “0.5-5” (ref) and “<0.5 or >5”. Lastly, Nt-proBNP was classified in pg/mL as “<450” (ref) and “≥450”. We inferred missing values with normal (reference) values. All statistical analyses were performed with R software (R Foundation for Statistical Computing).

In line with the French, Danish, and Bulgarian legislations on reuse of deidentified data collected during routine medical care, approval by one or more institutional review boards was not required. The study procedures complied with principles outlined in the Declaration of Helsinki.

The overall study database included 156,893 inpatient stays, of which the 4047 (2.58%) with VKA administration were analyzed. The mean age (Figure 2) was 75.9 years (SD 12.0), and there were 2356 women (58.2%).

The median length of stay was 9 days (IQR 6-15), and there were 162 in-hospital deaths (4.00%). The VKA administered was fluindione in 3256 cases (80.5%), warfarin in 553 cases (13.7%), acenocoumarol in 227 cases (5.6%), and another VKA or several different VKAs in 11 cases (0.3%).

For some DDI rules, fewer than 3 cases of concomitant administration with a VKA were observed in the database, so we did not compute the ORs. The corresponding drugs were as follows:

There were 76 drugs analyzed upon initiation:

There were 106 drugs analyzed upon discontinuation:

For other drugs, at least 3 cases of concomitant administration with a VKA were observed.

Upon initiation, 47 drugs did not appear to have a statistically significant impact on the INR:

Upon discontinuation, 14 drugs did not appear to have a statistically significant impact on the INR:

The results of the DDI rules for which at least one OR was significant are summarized in Table 1 (for drug initiation) and Table 2 (for drug discontinuation). The “n” column always refers to the number of stays with a VKA and the given drug, although the OR was always estimated for 4047 stays. All of the drugs evaluated in the tables were associated with a protective effect.

In this study, all the drugs that reportedly interact with VKAs either lacked a statistically significant association or were associated with a statistically significant reduction in risk. Our results suggest that an empirical evaluation of DDIs (as has been suggested for an SPC-CDSS) could help to refine the alerts issued by a CDSS [34]. Our objective was to determine which drugs were associated with an increased risk of bleeding or thrombosis (compared with baseline), rather than to discover which drugs indeed interact with VKAs. It should also be borne in mind that the risk baseline was not zero but corresponded to the actual risk to which inpatients in a given hospital were exposed. This risk was already quite high, and the purpose of a CDSS is to warn physicians when this risk will be accentuated. Hence, our present findings do not contradict the current body of academic knowledge about these drugs.

In all included hospitals, various CDSSs were active before the time of the study. In all of them, the physicians asked for all the alerts to be deactivated. Indeed, physicians were under alert fatigue. Those bad experiences led them to set up the PSIP European Project [9], whose purpose was to find “intelligent” ways to prevent adverse drug events. This paper stands in continuation of the PSIP Project.

Our study had several strengths. First, the drugs for evaluation were identified through a systematic review of the literature. Second, the study was population-based; in contrast to clinical trials, it was possible to analyze real-life drug administrations, ill-advised drug combinations, and patients with several comorbidities. Along with the INR values, we also took account of the chronology of the drug prescriptions and discontinuations.

Our observational study also had several limitations. First, the number of exposed patients was too small for many drugs. Consequently, our study was not powerful enough to provide firm evidence of an increase or a decrease in the probability of outcomes. This limitation highlights the shortcomings of the SPC-CDSS concept. A reasonable attitude would be to ignore statistical filtering when the number of cases in the learning database is too small. Second, the dose levels of the drugs involved in the DDIs have not been evaluated. Therefore, it cannot be excluded that patients were overdosed or underdosed, which could falsely affect our results. Third, polypharmacy was common (especially in the elderly population; the mean age was 75.9 years) but could not be fully taken into account. Therefore, an outcome counted for one DDI rule could potentially be due to another DDI rule being administered concomitantly to the patient. Fourth, we considered that data were not missing at random and so imputed missing data with normal values; in routine clinical care, nonmeasured parameters are more likely to be normal. Lastly, we used the same onset time (1 day) and discontinuation time (4 days) for each drug, even though the pharmacokinetics differed. Naturally, pharmacokinetics of other possibly interacting drugs are not similar: some of them have a short half-life, and others have a long half-life. Moreover, the kinetics of the interaction cannot be directly inferred from the half-life. Taking this into account would require having a precise description of the mechanisms of all interactions, which is not possible.

The INR is a surrogate marker and does not necessarily reflect clinical outcomes. Indeed, a high INR does not always result in bleeding, nor does a low INR in thrombosis. Furthermore, some DDI interactions for VKAs may lead to clinical outcomes without any change in the INR. However, these clinical outcomes would not have been measured as frequently as the INR was, and the measurements would have been less reliable. Although this would be an issue in automated ADE detection, this approximation is still acceptable when the objective is to filter alerts and identify risk factors.

The number of different patients was 3101 for 4047 stays. Correlation between patients was not taken into account. This attitude can be justified as follows. The calibration of the CDSS is carried out based on statistical individuals that correspond to solicitations of the inference engine and not to physical persons. If some specific patients are more often hospitalized, it makes sense to overweight their statistical properties in the CDSS.

The statistically significant associations observed for some drugs should not be interpreted as proof of a causal relationship. Indeed, many drugs are associated with specific clinical contexts (ie, indication bias). Those contexts are variously related to the patient (eg, treatments for Alzheimer disease and age), the context of care (eg, antibiotics and bacterial infection), or the prescriber (eg, a cardiologist who is used to prescribing VKAs and avoids DDIs). It should be noted that our present results do not cast doubt on our current body of pharmacological knowledge per se; however, they do challenge the commonly accepted idea whereby this knowledge alone should be used to filter or rank DDI rules [20,21,27]. We suggest that “real-life” empirical probabilities might be more appropriate for these purposes: an alert should be flagged up because there is an actual ADE risk (considering the context, ie, confounding factors, the patient, and the prescriber) and not only a theoretical risk. Perhaps the root of the problem is not so much the DDIs, but the pathological context of the patient. Our hypothesis is that for patients who are doing well, DDIs have a relatively limited impact, due to physiological adaptability. On the other hand, for patients with multiple comorbidities, DDIs have a stronger impact [43,44]. However, using empirical probabilities to automatically filter or rank DDI rules raises a number of issues; the probabilities would have to be updated frequently and computed separately in various contexts [35].

These probabilities could be used to improve CDSSs in two ways, both of which have been suggested and tested in the literature [35,36,45]: first, to deactivate DDI rules that are associated with an empirical probability below a chosen threshold, and second, to show physicians past cases with outcome to improve their adherence to remaining alerts. The SPC-CDSS concept was recently introduced [34]. The idea is to automatically reuse actual clinical data and search for outcomes (INR≥5, for instance). To prevent the occurrence of an outcome, the SPC-CDSS automatically estimates the conditional probability of an outcome for each rule, assuming that its conditions are met. When the probability is too low (and if there are enough patients), the corresponding alerts are automatically deactivated. In our present work, we used a type 1 error of 5%. A higher threshold (eg, 10%) would remove fewer alerts. The threshold could then be tuned according to the individual physician’s level of risk aversion and alert tolerance. This calculation could also be performed separately for each medical specialty, to take account of the context. This could include latent variables (eg, mean patient characteristics, comorbidities, and the reason for admission), organizational characteristics, and physician characteristics.

As reported in the literature [36,46,47], our present findings confirmed that the reuse of EHR data is an effective way of identifying likely ADEs. Indeed, active postmarket surveillance of drugs must be based on the reuse of data from EHRs and, more specifically, on the inpatient setting; the latter has not been extensively studied [48].

After calculating the probability that specific medications would interact with VKAs in real life, we found that many of the medications did not show the predicted DDIs. We suggest that EHR data can be automatically mined to filter DDI rules and thus improve CDSSs.