Quality Metrics for Linked Open Data

Editor's Notes

• #4 The main goal of LOD is to create knowledge by interlinking already existing, but dispersed data. LOD has built on three principles: URI, RDF, HTTP
• #5 To better understanding of the problem, we review the requirements for publishing a datasets. First, it should be resolvable by URI. However, the usefulness of knowledge created through LOD, strongly depends on the quality of the published data. Now, a question. If a dataset satisfy all of these criteria, is it ready to publish? This is the question that we are going to answer now.
• #6 As the main idea of linked data is based on “publishing first, improving later”, it leads to some quality problems in published datasets. On the other hand, the usefulness of the knowledge created from LOD, strongly depends on the quality of its data, so there is a missing link in publishing process. It is DATA Quality evaluation before Release
• #8 So, we have proposed an automated and scalable method using metrics, to assess the quality of datasets before publishing and interlinking to LOD. To the best of our knowledge, we found that many of the published datasets suffer from quality issues. We believe that most of these issues have roots in the deficiencies of the sources from where that data is extracted, and they can be avoided if they are identified in the initial stages of publishing
• #9 We present the quality deficiencies of data within the published datasets focusing on those which are related to the dataset itself and also detectable in the initial phase of publication. To this end, we have classified the quality issues into schema level and instance level as presented in these tables.
• #10 In summary, we have identified eleven quality deficiencies characterizing eighteen quality issues at both schema and instance levels. Among all, six quality issues, which their resolution methods are domain expert or ontologist, cannot be detected by any kind of automated methods and needs the intervention of human experts. It is clear that all of these metrics are very subjective and it is hard, if not impossible, to asses them automatically. Among the remaining quality issues, only three issues can be detected and resolved by validators. To the extent of our knowledge, there is no validator to cover all of the remaining issues, particularly the issues relating to incompatibility of schema, naming, and inconsistent data. Thus, we propose a set of metrics to address the remaining quality issues. We note that the identified issues and the following proposed metrics are not meant to be comprehensive and are only limited to the issues reported in the literature.
• #11 In this section, a set of metrics are proposed to address mentioned quality issues which can be resolved in an automated or semi-automated way. To achieve this, metrics, proposed in the areas of the Linked Data, relational databases, and data quality models have been considered; the results of which were taken into account as guidelines for designing a useful set of metrics for our purpose. The proposed metrics are presented in this table. According to the definitions presented for the metrics, it is clear that all of the metrics are defined to measure the quality problems in the scope of the RDF dataset itself, not in the context of other datasets. From the level of quality deficiency point of view, the last two metrics (Im_DT and Sml_Cls) are defined to address intrinsic quality issues at the schema level, while the others are related to the intrinsic quality problems at the instance level. All of the proposed metrics are theoretically validated in our recently published work [14].
• #12 It is necessary to place metrics under empirical evaluation to observe their behavior and show their applicability in practice. So, we first calculated the values of the metrics for eight datasets of different domains and sizes from LOD. Next, we manipulated the quality of these datasets by applying some heuristics and then recalculated the metric values to observe the behavior of the metrics over these changes. The aim of our dataset manipulation work was to investigate the trends of metrics over real datasets and to compare the results of applying metrics on good and poor quality data.
• #13 We have selected eight datasets from the EU FP6 Networked Ontology (NeOn) project.
• #14 We have implemented an automated tool that is able to automatically compute the metric values for any given input dataset.
• #15 Then we used Spearman test to study metric interdependency. According to Spearman’s correlation, a correlation with a significance p value ≤ 0.05 can be considered to be significant. Two pairs of correlated are metrics and the others are independent. The reason for the observed correlation is that the values for correlated metrics are mainly '0', and it indicates that there are not many deficiencies related to the mentioned metrics in the datasets.
• #16 Next, we manipulated the quality of these datasets by applying some heuristics and then recalculated the metric values to observe the behavior of the metrics over these changes. The aim of our dataset manipulation work was to investigate the trends of metrics over real datasets and to compare the results of applying metrics on good and poor quality data. To this end, fourteen heuristics are introduced to be used in the dataset contamination process. Some of these quality issues such as misspelling errors were made using an ontology editor, i.e. Protégé. For those quality issues such as invalid usage of inverse functional properties, errors were introduced manually. We have randomly applied the heuristics to the different datasets. The rationale for this was to measure the values for our metrics both before and after the quality issues were injected.
• #17 The aim of the second experiment is to show the trends of the proposed metrics by recalculating the values of the metrics over manipulated datasets. As a result, we would ideally expect to observe meaningful changes in the values of the metrics according to the heuristics used to manipulate the datasets. In light of the results, it is observed that most of the outcomes are desirable; however, some of the values need more discussions.
• #18 Regarding to the results, we observed that in 78% of the scenarios the metrics have behaved as expected. Dependency between heuristics lead to some side effects Some of the heuristics were not independent and as a result, the order of applying these heuristics affected the results of measuring the quality problems by the metrics. The ratio of heuristics done over the size of datasets is very low Based on this experiment, whenever the number of changes is less that 10% of the triples, the changes of metric values cannot be properly reported. Although in our scenario, no radical shift in the metric values was observed, but we are not going to generalize our finding about the trends of metrics, because of the limited number of datasets that we have used in this experiment. As a result, we believe more experiments need to be done to reach a valid conclusion about the reaction of metrics to the changes in the measurement subject.