According to our heuristic rules, chemical D007545 in sentence 4 and disease D006965 in sentence 5 are regarded as an intersentence level instance because there are no mentions of them in the same sentence. Chemical D007545 in sentence 1 and disease D006965 in sentence 5 will be omitted because their distance is not the shortest. Further, chemical D007545 in sentence 4 and disease D006984 in sentence 5 are not regarded as an intersentence level instance because chemical D007545 already has intrasentence level co-occurrence with disease D006984 in sentence 4.

We used ELMo instead of the traditional word representation in our model. Traditional word representation generates a fixed representation vector for the same word. However, ELMo is the function of the entire input sentence based on a bidirectional language model so that it can generate different representation vectors for the same word according to the different sentence context.

Given that a sequence {t₁, t₂,…..t_N} denotes the word tokens in a sentence S. Given a token t_k, the forward language model calculates the probability of the token t_k based on the previous tokens {t₁, t_2,..., t_(k-1)} of t_k in the sentence S as follows:

(1)

Similarly, the backward language model calculates the probability of the token t_k based on the back tokens {t₁, t_{2, …,} t_(k-1)} of t_k in the sentence S as follows:

(2)

Combining the forward and the backward language models as a bidirectional language model, the log-likelihood can be maximized as follows:

(3)

ELMo can represent the semantic and syntactic information of the word. In our model, we use a linear combination of the hidden state in each layer of the bidirectional language model to generate a deep contextualized representation for words. The POS and the position information of a word are crucial to biomedical relation extraction. Therefore, we also utilize POS embedding and position embedding to enhance the representation ability of the input. The POS embedding represents the POS feature of a word, and the position embedding reflects the relative distance between the word and the target entity. Given a word at position i, we obtain its POS embedding w_p,i and position embedding w_d,i based on mapping matrixes M_p and M_d, respectively. Finally, the whole word representations concatenate deep contextualized word representations, POS embedding, and position embedding as follows:

The LSTM model is a variant of recurrent neural network models that has been used in many NLP tasks successfully. The LSTM model overcomes the vanishing gradient problem by introducing a gating mechanism [24]. Therefore, it is suitable to capture the long-term dependency feature. The LSTM unit consists of 3 components: the input gate i_t, the forget gate f_t, and the output gate o_t. At the time step t, the LSTM unit utilizes the input word x_t, the previous hidden state h_(t-1), and the previous cell state c_(t-1) to calculate the current hidden state h_t and cell state c_t. The equations are as follows:

where W, U, b are the weight and bias parameters, and ⊙ denotes element-wise multiplication. In this study, we use the BiLSTM model that can capture the forward and backward context features simultaneously. The BiLSTM model combines a forward LSTM and a backward LSTM. Given the hidden state of the forward LSTM and the hidden state of the backward LSTM , the final hidden state is concatenated as:

The BiLSTM model learns the context features from the input sequences automatically and effectively. However, these features make different contributions to the biomedical relation extraction. In our model, we capture the relatively important features by introducing multihead attention mechanism. The essence of multihead attention is applying self-attention mechanism multiple times so that it may let the model learn the relatively important features from different representation subspaces. The self-attention mechanism generates the output based on a query and a set of key-value pairs. The output is the weighted sum of the values, where the weight assigned to each value is computed by applying attention function to the query with the corresponding key. In our study, we deal with the output of the BiLSTM model by multihead self-attention. Further, we use the dot-product attention function instead of the standard additive attention function [25] as follows:

(11),

where Q, K, V∈Rⁿ represent query, key, and value matrixes, respectively. d is the dimension of the output of the BiLSTM model.

The main idea of the multihead attention is applying the self-attention mechanism multiple times. If the multihead attention contains h heads, the i-th attention head can be calculated as head_i=Attention (Q_i,K_i,V_i). Thus, the final multihead attention is the concatenation of {head_1,head_2,...,head_h} as MultiHead (Q,K,V)=Concat (head_1,head_2,...,head_h) W^o. The output of the multihead attention layer is a matrix of R^nat.

GCN is an adaptation of CNN [26], which operates on graphs. Given a graph with n nodes, the graph structure can be represented as an adjacency matrix A. In this study, we converted the document-level dependency graph into its corresponding adjacency matrix A, where A_ij=1 if there is a dependency edge going from token i to token j; otherwise A_ij=0. The dependency graph can be calculated as an undirected graph [27], which means A_ij=A_ji. Further, we add a self-node edge to all the nodes in the graph, which means A_ii=1. Since the degree of a node in the dependency graph varies a lot, this may bias the output representation toward favoring high-degree nodes, regardless of the information carried in the node. To solve this issue, we normalize the activations in the graph convolution before feeding it through the nonlinearity. Finally, the graph convolution operation for node i at the l-th layer where and denote the input representation and the output representation of node can be defined as follows:

(12),

where W^(l) is the weight matrix, b^(l) is the bias vector, is the degree of node i in the dependency graph, and ρ is an activation function (eg, a rectified linear unit).

The GCN model takes the output of the BiLSTM model as the input word representation: Then, we stack the graph convolution operation over layers and obtain as the output word representations of the GCN model. Note that the GCN model presented above uses the same parameters for all edges in the dependency graph.

To make use of the output word representation of the GCN model for relation extraction, we generate the sentence representation as follows:

where h^(L) denotes the output representations at the last layer L of the GCN model, and f:R^n:R→R^d is a max-pooling function that maps n output vectors to the sentence vector.

Inspired by recent studies [28,29], entity information is central to relation classification. Therefore, we also obtain the chemical entity representation h_c as shown in . Similarly, we can obtain the disease entity representation h_d. The feature representation of the whole GCN model is h_GCN=[h_sent; h_c; h_d].

We also obtain the feature representation h_att from the output of the multihead attention layer by applying max pooling to the multihead attention matrix. We concatenate h_GCN and h_att to form the final representation h_final=[h_GCN; h_att] for relation classification. Then, the final representation is fed into a 2-layer perceptron as follows:

(14) and (15).

Finally, the hidden representation h₂ is fed to a Softmax function to calculate the confidence of the CID relation:

where o is the output, W_o is the weight matrix, and b_o is the bias vector.

We evaluated the effectiveness of the input representation of our model. We used the same model that we proposed and changed the input representations. The comparison performance of the different input representations is presented in Table 2.

In Table 2, we can observe that the model achieves an F-measure of 57.0% when we only use the pretrained word embedding as the input representation. When we concatenate the pretrained word embedding and position embedding, the F-measure is improved from 57.0% to 58.2%, which yields a 1.2% improvement. When we concatenate the pretrained word embedding, position embedding, and POS embedding as the input representations, we yield another 1.9% improvement compared with only using the pretrained word embedding and position embedding. The result indicates that both POS and position features are effective for the CID relation extraction. The deep contextualized word representation ELMo significantly outperforms the pretrained word embedding and yields a 4.8% improvement in the F-measure. The result indicates that ELMo can generate a more comprehensive representation for the word according to the sentence context, which results in a better CDR performance. Similarly, combining the position and POS embedding with the deep contextualized word representation can further improve the performance. When we concatenate the deep contextualized word representation, position embedding, and POS embedding as the input representation, we achieve the best F-measure of 63.5%. We also use the word representations generated from the bidirectional encoder representations from transformers for biomedical text mining in a feature-based approach and achieve an F-measure of 63.4%, which is similar to using ELMo.

We evaluated the effectiveness of the multihead self-attention mechanism. We used the same model architecture that we proposed, but we dealt with the output of BiLSTM by different attention mechanisms. The attention mechanism is divided into 2 categories: single-head attention mechanism and multihead attention mechanism. In single-head attention mechanism, we use 3 types of attention function: additive attention, general attention, and scaled dot-product attention, as shown below.

(17)

(18)

(19)

where h_i is the output of the BiLSTM, W₁, W₂, s, v are the parameter matrixes, and d is the dimension of the output of the BiLSTM model. The formula of the multihead attention is described in formula (11). The comparison performance of the different attention mechanism is presented in Table 3.

In Table 3, we can see that using the attention mechanism can improve the performance of the CID relation extraction. The multihead attention mechanism is more helpful than other single-head attention mechanisms. This suggests that the multihead attention mechanism can capture more valuable features from different representation subspaces.

We evaluated the effectiveness of the number of heads of the multihead attention mechanism. In this comparative experiment, we used the deep contextualized word representation, position embedding, and POS embedding as the input representation, and the dimensions of query, key, and value are the same. As shown in Table 4, we only varied the number of heads of the multihead attention.

In Table 4, we can see that the multihead attention mechanism can effectively improve the performance of the CID relation extraction. We can observe that the F-measure ranges from 62.2% to 63.5% when setting a different number of heads. When the number of heads is too little or too large, the performance will drop off. In short, we achieve the best F-measure of 63.5% when we set the number of heads as 5.