3.1. Research Approach

This study proposes a method for extracting a set of identical entities from heterogeneous knowledge sources. An identical relationship of entities is based on calculating the properties and its values of the entities. The analysis is performed through a combination of several methods called ‘strategy.’ In this paper, five strategies are introduced and are combined for extracting and verifying identical relationships of entities. Each strategy has its own advantages and disadvantages. For example, a consistency strategy is a simple method for extracting entities, but it returns high ambiguities as noise to some extent, whereas a max confidence strategy delivers reduced ambiguities by calculating a confidence score of entity pairs. Although the max confidence method would be useful for extracting entity pairs compared to the consistency method, the max confidence strategy is based on the entity pairs extracted by the consistency one. Therefore, each strategy can be used for individual purposes, and also can be applied to determine a high quality of identical entity pairs by combining several strategies.

3.2. A Formal Model of an Entity Pair

Let knowledge bases K₁ and K₂ contain a set of entities and properties, respectively. The set of entities is ${\mathit{\text{K}}}_i^{\text{E}}$ = { ${\mathit{\text{K}}}_i^{e_1}$, ... , ${\mathit{\text{K}}}_i^{e_n}$} and the set of properties in K_i is ${\mathit{\text{K}}}_i^p$ = { ${\mathit{\text{K}}}_i^{p_1}$, ... , ${\mathit{\text{K}}}_i^{p_n}$}. In addition, let ${\mathit{\text{K}}}_i^o$ = { ${\mathit{\text{K}}}_i^c$, ... , ${\mathit{\text{K}}}_i^p$} be the ontology schema of K_i, where ${\mathit{\text{K}}}_i^c$ is the set of classes and ${\mathit{\text{K}}}_i^p$ is the set of properties. Thus, entity pairs ${\mathit{\text{EP}}}_{\mathit{\left(}{\mathit{\text{K}}}_{\mathit{1}\mathit{,}}{\mathit{\text{K}}}_{\mathit{2}}\mathit{\right)}}$ as a set of identical entities for given knowledge bases K₁ and K₂ are denoted as follows:

${\mathit{\text{EP}}}_{\mathit{\left(}{\mathit{\text{K}}}_{\mathit{1}\mathit{,}}{\mathit{\text{K}}}_{\mathit{2}}\mathit{\right)}}\mathit{=}\mathit{\left\{}\mathit{\right(}{\mathit{\text{K}}}_{\mathit{1}}^{e_{\mathit{1}}}\mathit{,}{\mathit{\text{K}}}_{\mathit{1}}^{e_j}\mathit{)}\mathit{,}\mathit{.}\mathit{.}\mathit{.}\mathit{,}\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}^{e_s}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_t}\mathit{\left)}\mathit{\right\}}$

where ${\mathit{\text{K}}}_i^{e_1}$ is identical to ${\mathit{\text{K}}}_{\mathit{2}}^{e_j}$. On the other hand, the schema alignment K^o is aligned to its schemas:

${\mathit{\text{K}}}^o={\mathit{\text{K}}}_{\mathit{1}}^o\stackrel{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}}{\iff }{\mathit{\text{K}}}_{\mathit{2}}^o$

where ${\mathit{\text{K}}}_{\mathit{1}}^c\stackrel{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}}{\iff }{\mathit{\text{K}}}_{\mathit{2}}^c$ is the class alignment and ${\mathit{\text{K}}}_{\mathit{1}}^p\stackrel{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}}{\iff }{\mathit{\text{K}}}_{\mathit{2}}^p$ is the property alignment for K₁ and K₂. In this sense, ${\mathit{\text{K}}}_{\mathit{1}}^{c_i}\stackrel{align}{\iff }{\mathit{\text{K}}}_{\mathit{2}}^{c_j}$ means that ${\mathit{\text{K}}}_{\mathit{1}}^{c_i}$ is identical to ${\mathit{\text{K}}}_{\mathit{2}}^{c_j}$, and ${\mathit{\text{K}}}_{\mathit{1}}^{p_i}\stackrel{align}{\iff }{\mathit{\text{K}}}_{\mathit{2}}^{p_j}$ means that the value of P_i in K₁ corresponds to that of P_j in K₂. Thus, according to K^o, a set of property mappings to the matching keys is defined as follows:

${\mathit{\text{MK}}}_{\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}\mathit{)}}\mathit{=}\mathit{\left\{}\mathit{\right(}{\mathit{\text{K}}}_{\mathit{1}}^{p_i}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{p_j}\mathit{)}\mathit{,}\mathit{.}\mathit{.}\mathit{.}\mathit{,}\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}^{p_s}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{p_t}\mathit{\left)}\mathit{\right\}}$

4.1. Consistency Strategy

This strategy aims to extract a set of precise entities by mapping property values on specific knowledge bases. That is, to determine the consistency of ${\mathit{\text{K}}}_{\mathit{1}}^{e_i}$ and ${\mathit{\text{K}}}_{\mathit{2}}^{e_j}$ based on matching keys MK, two strategies, S_I and S_U, are defined:

Strategy S_I: For ${\mathit{\text{K}}}_{\mathit{1}}^{e_m}$ and ${\mathit{\text{K}}}_{\mathit{2}}^{e_n}$ from K₁ and K₂, $\forall ({\mathit{\text{K}}}_{\mathit{1}}^{{\mathit{\text{P}}}_i}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{{\mathit{\text{P}}}_j})\in {\mathit{\text{MK}}}_{\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}\mathit{)}}$ , the ${\mathit{\text{K}}}_{\mathit{1}}^{p_i}$ value of ${\mathit{\text{K}}}_{\mathit{1}}^{e_m}$ is exactly equal to the ${\mathit{\text{K}}}_{\mathit{2}}^{p_j}$ value of ${\mathit{\text{K}}}_{\mathit{2}}^{p_j}$. Then, $\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}^{e_m}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_n}\mathit{)}$ is an identical entity pair, and the consistency determination is of the intersection strategy S_I.

Strategy S_U: For ${\mathit{\text{K}}}_{\mathit{1}}^{e_m}$ and ${\mathit{\text{K}}}_{\mathit{2}}^{e_n}$ from K₁ and K₂, $\exists ({\mathit{\text{K}}}_{\mathit{1}}^{{\mathit{\text{P}}}_i}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{{\mathit{\text{P}}}_j})\in {\mathit{\text{MK}}}_{\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}\mathit{)}}$, the ${\mathit{\text{K}}}_{\mathit{1}}^{p_i}$ value of ${\mathit{\text{K}}}_{\mathit{1}}^{e_m}$ is exactly equal to the ${\mathit{\text{K}}}_{\mathit{2}}^{p_j}$ value of ${\mathit{\text{K}}}_{\mathit{2}}^{e_n}$. Then, $\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}^{e_m}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_n}\mathit{)}$ is an identical entity pair, and the consistency determination is of the union strategy S_U.

This strategy is based on the assumption that all knowledge sources are trustworthy: The knowledge in K_i is precise and without defect. The identical relations ${\mathit{\text{EP}}}_{\mathit{\left(}{\mathit{\text{K}}}_{\mathit{1}\mathit{,}}{\mathit{\text{K}}}_{\mathit{2}}\mathit{\right)}}$ extracted by this strategy are considered precise because the mapping of the property values is exact without bias. On the contrary, most open knowledge bases contain some defects which may be caused by false recognition, inaccurate source, or knowledge redundancy. Note that one entity can be interlinked to multiple entities of different knowledge sources (e.g. $\exists \mathit{(}{\mathit{\text{K}}}_{\mathit{1}}^{e_i}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_j}\mathit{)}\mathit{,}\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}^{e_s}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_t}\mathit{)}\in {\mathit{\text{EP}}}_{\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}\mathit{)}}$, and ${\mathit{\text{K}}}_{\mathit{1}}^{e_i}$ = ${\mathit{\text{K}}}_{\mathit{1}}^{e_s}$ and ${\mathit{\text{K}}}_2^{e_j}\ne {\mathit{\text{K}}}_2^{e_t}$). This ambiguous pair might arise from a defect in the knowledge base K_i. For establishing highquality linkages across heterogeneous knowledge sources, it is essential to extract confident ${\mathit{\text{EP}}}_{\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}\mathit{)}}$ by eliminating ambiguities to the greatest extent possible. Therefore, alternative strategies are proposed.

4.2. Max Confidence Strategy

This strategy calculates a confidence score for the entity pairs extracted by the consistency strategy to reduce the noise caused by defects and determines precise and confident entity pairs. The formal notation of this strategy is defined as follows:

Given matching keys ${\mathit{\text{MK}}}_{\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}\mathit{)}}\mathit{=}\mathit{\left\{}\mathit{\right(}{\mathit{\text{K}}}_{\mathit{1}}^{p_i}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{p_j}\mathit{)}$, ... , $\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}^{p_s}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{p_t}\mathit{)}\mathit{\}}$, for ${\mathit{\text{K}}}_{\mathit{1}}^{e_m}\in {\mathit{\text{K}}}_{\mathit{1}}^{\mathit{\text{E}}}$ and ${\mathit{\text{K}}}_{\mathit{2}}^{e_n}\in {\mathit{\text{K}}}_{\mathit{2}}^{\mathit{\text{E}}}$, let ${\mathit{\text{MK}}}_m\mathit{=}\{{\mathit{\text{K}}}_{\mathit{1}}^{p_{im}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{p_{jm}}\mathit{)}\mathit{,}\mathit{.}\mathit{.}\mathit{.}\mathit{,}\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}^{p_{sm}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{p_{tm}}\mathit{)}\}$ be the matched ${\mathit{\text{MK}}}_{\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}\mathit{)}}$, where $({\mathit{\text{K}}}_{\mathit{1}}^{p_{im}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{p_{jm}})$ indicates that the ${\mathit{\text{K}}}_{\mathit{1}}^{p_{im}}$ value of ${\mathit{\text{K}}}_{\mathit{1}}^{e_m}$ is exactly equal to the ${\mathit{\text{K}}}_{\mathit{2}}^{p_{jm}}$ value of ${\mathit{\text{K}}}_{\mathit{2}}^{e_n}$. Based on this, ${\mathit{\text{MK}}}_m$ and ${\mathit{\text{MK}}}_{\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}\mathit{)}}$ can be defined as ${\mathit{\text{MK}}}_m\subseteq {\mathit{\text{MK}}}_{\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}\mathit{)}}$, then a confidence score of $({\mathit{\text{K}}}_{\mathit{1}}^{e_m}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_n})$ is calculated by the following equation:

$conf({\mathit{\text{K}}}_{\mathit{1}}^{e_m}\mathit{,}{\mathit{\text{K}}}_{\mathit{1}}^{e_n})=\parallel {\mathit{\text{MK}}}_m\parallel /\parallel {\mathit{\text{MK}}}_{\mathit{\left(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{\text{,}}{\mathit{\text{K}}}_{\mathit{2}}\mathit{\right)}}\parallel$

where $\parallel \cdot \parallel$ is the cardinality. Therefore, a confidence score is assigned to each entity pair, and for $({\mathit{\text{K}}}_{\mathit{1}}^{e_i}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_j})\text{ , }({\mathit{\text{K}}}_{\mathit{1}}^{e_s}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_t})\in {\mathit{\text{EP}}}_{\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}\mathit{)}}$. Therefore, $({\mathit{\text{K}}}_{\mathit{1}}^{e_i}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_j})$ is the confident identical entity pair where $conf({\mathit{\text{K}}}_{\mathit{1}}^{e_i}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_j})\mathit{\text{>conf}}({\mathit{\text{K}}}_{\mathit{1}}^{e_s}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_t})$.

4.3. Threshold Filtering Strategy

The Max Confidence allows to filter out ambiguous same entity pairs; nonetheless, some of entity pairs may have relatively high scores with low confidence levels. To solve this issue, a threshold is added to the extraction process: If an entity pair has determined with the highest confidence and it has a low score compared to other scores, it can be removed from a set of candidates. The threshold filtering strategy aims to improve a confidence level of extracted entity pairs by using a threshold score. Given a threshold $\mathit{\text{θ}}$ for $({\mathit{\text{K}}}_{\mathit{1}}^{e_i}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_j}),({\mathit{\text{K}}}_{\mathit{1}}^{e_i}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_t})\in {\mathit{\text{EP}}}_{\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}\mathit{)}}$, where $conf({\mathit{\text{K}}}_{\mathit{1}}^{e_i}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_j})\mathit{\text{>conf}}({\mathit{\text{K}}}_{\mathit{1}}^{e_i}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_t})$ and $conf({\mathit{\text{K}}}_{\mathit{1}}^{e_i}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_j})\ge \mathit{\text{θ,}}({\mathit{\text{K}}}_{\mathit{1}}^{e_i}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_j})$ is selected as the confident same entity pair.

4.4. One-to-One Mapping Strategy

This strategy extracts simply 1-1 entity pairs from heterogeneous knowledge sources by ignoring multiple relations in which one identifier is matched to multiple identifiers of different sources. Formally, it is represented as $\forall ({\mathit{\text{K}}}_{\mathit{1}}^{e_i}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_j})\in {\mathit{\text{EP}}}_{\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}\mathit{)}},\nexists ({\mathit{\text{K}}}_{\mathit{1}}^{e_s}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{e_t})\in {\mathit{\text{EP}}}_{\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}\mathit{)}}$ where i = s or j = t. By applying one-to-one mapping, identical entity pairs ${\mathit{\text{EP}}}_{\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}\mathit{)}}$ have no ambiguous relations.

4.5. Belief-based Strategy

The four strategies introduced so far focus on interrelations between entity pairs by comparing properties of knowledge bases, whereas they do not consider intrarelations in a certain pair. In other words, property values of entities in a certain pair should be checked for determining identical relations. The belief-based strategy aims to analyse property values of extracted entity pairs that is based on the Dempster-Shafer theory (Yager, 1987), also called the theory of evidence.

Given a set of same entity pairs EP, let X_EP denote the set representing all possible states of an entity pair. Here, two cases are possible: The two entities are linked (L) or the two entities are not linked (U). Note that X_EP = {L, U}. Then, ${\mathit{\text{Ω}}}_{{\mathit{\text{X}}}_{\mathit{\text{EP}}}}$= { $\mathit{\text{Φ}}$ , L, U, {L, U}}, where indicates the empty set, and {L, U} indicates that it is uncertain whether $\mathit{\text{Φ}}$ they are linked. Therefore, a belief degree is assigned to each element of ${\mathit{\text{Ω}}}_{{\mathit{\text{X}}}_{\mathit{\text{EP}}}}$:

where m is the degree of same belief, which is the basic belief assignment in the Dempster-Shafer theory. Then, each pair of knowledge sources has four hypotheses, and the formal model is represented as follows:

Given ${\mathit{\text{MK}}}_{\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}\mathit{)}}=\left\{\right({\mathit{\text{K}}}_{\mathit{1}}^{p_i}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{p_j}),...,({\mathit{\text{K}}}_{\mathit{1}}^{p_s}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{p_t}\mathit{\left)}\right\}$ and ${\mathit{\text{MK}}}_{\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}\mathit{)}}=\left\{\right({\mathit{\text{K}}}_{\mathit{1}}^{p_{im}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{p_{jm}}),...,({\mathit{\text{K}}}_{\mathit{1}}^{p_{sm}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}^{p_{tm}}\mathit{\left)}\right\}$, m is assigned as follows:

where ${\mathit{\text{MK}}}_{um}$ represents the unmatched ${\mathit{\text{MK}}}_{\mathit{(}{\mathit{\text{K}}}_{\mathit{1}}\mathit{,}{\mathit{\text{K}}}_{\mathit{2}}\mathit{)}}$, that is, the ${\mathit{\text{K}}}_{\mathit{1}}^{p_i}$ value of ${\mathit{\text{K}}}_{\mathit{1}}^{e_m}$ is not equal to the ${\mathit{\text{K}}}_{\mathit{2}}^{p_j}$ value of ${\mathit{\text{K}}}_{\mathit{2}}^{e_n}$. And uncertain pairs of knowledge sources are calculated by the following model:

According to the theory of evidence, the basic belief assignment m(A), $\mathit{\text{A}}\in \mathit{\text{Ω}}$ , expresses the proportion of all relevant and available evidence that supports the claim that the actual state belongs to A. In this sense, a degree of belief is represented as a belief function rather than a Bayesian probability distribution.

6.1. Data Collection

Two knowledge bases (i.e., Wikidata and Freebase) are selected to demonstrate the proposed strategies. Wikidata and Freebase are receiving great attention from academia and industry for constructing their own knowledge bases, and there are realistic issues for data integration between two knowledge sources. It is essential to derive homogeneous entities for knowledge integration, since Wikidata and Freebase have been developed independently. A set of same entities between Freebase (2015-02-10)⁵ and Wikidata (2015-02-07) is extracted via their own Wikipedia reference links (i.e., wiki-keys of Freebase and Wikipedia URLs of Wikidata). After pre-processing the collected datasets, 4,446,380 entities from Freebase and 15,403,618 entities from Wikidata are extracted with Wikipedia links. By using the consistency strategy (i.e., S1), 4,400,955 pairs are obtained from both knowledge sources.

6.2. Results

The aim of applying different approaches for same extraction is to generate links with the highest confidence between Freebase and Wikidata entities. The results differed slightly with the given datasets. Fig. 3. The result of extracting same entities between Freebase and Wikidata.3 illustrates the results obtained using different mapping styles with the proposed strategies. Note that the consistency strategy obtains the largest number of entity pairs. Nonetheless, there are a number of 1-multiple/multiple-1/multiple-multiple links which cause ambiguities as shown in Table 3. Without applying any approaches, the consistency strategy possesses the largest ambiguity (0.37%). The oneto-one mapping obviously holds the full confident same entity mapping pairs. The Max Confidence, the Threshold Filtering (0.5 threshold), and the Belief-based strategies show great effect on elimination of ambiguity. The number of mapping pairs based on belief degree is approximated to that of Max Confidence. The belief degree greatly influences the reduction in ambiguity in the multiple Freebase case but not in the multiple Wikidata case.

새창으로 보기

Fig. 3.

The result of extracting same entities between Freebase and Wikidata.

As shown in Table 4, the precision and F1 score are 100 percent for all strategies, because the set of matching pairs is extracted by using the Strategy SU, whereas both the precision and F1 score are greater than 98.1371 percent, and precision scores are slightly differed among these strategies. Based on this result, a combination of each strategy can reduce some ambiguities that are not removed using a single approach. On the other hand, both the precision and F1 score of the belief-based strategy are 99.1165 and 99.5563, respectively. This demonstrates that the belief-based strategy provides an extremely high matching quality.

새창으로 보기

Table 3.

Composition of mapping results based on different strategies

새창으로 보기

Table 4.

Matching quality of proposed strategies

Note that Google has also constructed a mapping between Freebase and Wikidata that was published in October 2013. They detected 2,099,582 entity pairs with 2,096,745 Freebase entities and 2,099,582 Wikidata entities. Fig. 4 illustrates the result of identical entity pairs using the same datasets from Freebase and Wikidata. The entity pairs from all proposed strategies have some differences compared to the Google result. Although they did not explicitly announce how they extracted this result, it might use an exact matching of Wikipedia URL. Applying the proposed strategies to the Google results, the identical mapping pairs are more than 99.51%. However, they include ambiguous results according to individual strategies. For example, the consistency strategy has the highest different entities (1.59%), whereas the belief-based strategy is the smallest (0.45%). In summary, the belief-based strategy can be considered as an effective approach to reduce ambiguity for entity extraction. Note that matching performance of the Google result is not conducted, because they provided this dataset only once, and did not update related data sources.

새창으로 보기

Fig. 4.

A comparison of the Google result.