Data quality represents a very main challenge because the cost of anomalies can be very high especially for large databases in enterprises that need to exchange information between systems and integrate large amounts of data. Decision making using erroneous data has a bad influence on the activities of organizations. Quantity of data continues to increase as well as the risks of anomalies. The automatic correction of these anomalies is a topic that is becoming more important both in business and in the academic world [1] [2]. The data may be derived from different sources for which metadata can be totally absent and most often not sufficient to reflect the actual content of the data and treat any anomalies. Therefore, it is interesting to create new data integration tools to better understand the semantics and structure of the data. We develop this work in collaboration with the Talend Company which is an editor of one the most known of open source data integration and data quality tools [3]. The first part of the project consisted to treat the inter-lines anomalies [4]-[7]. Indeed, the automatic correction of anomalies in a same column gives better results, insofar as their semantics is known. Moreover, the discovery of semantic links that may exist between the columns may avoid the violation of various types of dependencies constraints between them. This is what constitutes the second part of the project which is the subject of this article. Constraint checking dependencies from large amounts of data will help to correct automatically the anomalies such as null values and some functional dependencies. In this paper, our proposal is to try to understand the semantics of the data before correcting it. An intra-column study allows the automatic correction of anomalies in the data related to its context. For example, the following two strings “Londres” and “London” are equivalent if they represent the context of city names but they are different if they are names of people. Algorithms of similarity distance calculation such as Levenshtein [8], Jaro-Winkler [9], Soundex [10] and Metaphone [11] do not take into account the context. Our approach is to illustrate the data structure (i.e. the semantic schema) of the data source. First, the syntactic and semantic automatic corrections inside one column can be better focused. Second, eventual semantic links that may exist between columns can be discovered. Then, the process of anomalies caused by the violation of dependency constraints will start. In a context of Big Data, we use the MapReduce technology to verify the discoveries dependencies and correct anomalies caused by the violation of these dependencies. The rest of the article is organized as follows. The second section describes the step of semantic recognition of data. Section three discusses the intra-column and inter-columns data cleaning step. Finally, we compare our approach with other work and we address our future goals.

The step of data categorization consists to determine the semantics of each column of a data source. Indeed, in order to qualify a syntactically incorrect data, it should be evaluated in its context. Several examples can illustrate this: 1) The string “Pari” can be considered syntactically incorrect only if it is the French name of the city “Paris”. The words “Pékin” and “Beijing” mean the same thing in two different languages if we know that these are names of cities. “Beijing” could be considered semantically incorrect if the used language (dominant language) is French; 2) The three strings “16-10-1996”, “10-16-1996” and “1996-16-10” may represent the same type of information “date with different formats” defined by a regular expression; 3) The string “1996-10” is not a date.

Then, we introduce the use of stored knowledge in a referential called Data Dictionary (DD) [5] [12] [13]. The DD contains knowledge identified by two modes: 1) Data defined by extension: a priori given list such as city names, country names and names of organizations. Valid Strings (DDVS) and Key Words (DDKW) are stored in the DD; 2) Data defined by intention: this knowledge must verify some properties such as regular expressions or belonging to an interval of values (DDRE). For example Emails, Websites and Dates must be conforming to a model which can be a regular expression. In other words, the DD can then be seen as a set of categories. Each category corresponds to only one data type (String, Number, Date). Some categories can have subcategories such as languages. We propose a DD that contains not only information grouped by categories (Valid Strings, Rules such as Regular Expression) but also semantically valid knowledge such as constraints or functional dependencies pre-stored. Figure 1 below shows examples of pre-stored knowledge.

We studied and compared the functionalities of some data quality tools such as Talend Data Quality [3] and Pentaho Data Integration [17]. These tools are used to verify only the functional dependencies given by the user. Then, the user must have knowledge about the data schema and the dependencies to be verified. These Tools do not correct anomalies caused by the violation of semantic dependencies between columns. They do not process null values. The different algorithms of discovery of functional dependencies [14] [15] [18] consist to find dependencies across all possible combinations for the different columns of a data source. This increases the size of the search space. These algorithms do not take into account the semantics of data, while, in our approach we focus on contextual data quality. The application of the principle of MapReduce in Big Data will allow to validate algorithms on large volumes of data with good perfermance. Few studies include the principle of MapReduce (distribution of data and treatments).

The goal of our work is to contribute to the development of new data integration tools in order to assist the user in the contextual data quality process. Our contribution is to understand the semantic of data before correcting them. In fact, our approach allows the categorization of data by giving them a category and eventually a subcategory. Semantic categorization allows to infer semantic links that can exist between the different columns. The recognition of the structure and semantics of data facilitates the detection and correction of various intra-column, inter-columns and inter-lines anomalies in the same data source. We propose a MapReduce algorithm of the verification of dependency constraints to detect anomalies caused by the violation of these constraints. We present an algorithm for the automatic correction of inter-columns anomalies especially the treatments of null values. The enrichment of the data dictionary will be the subject of our future work.

Houda Zaidi,Faouzi Boufarès,Yann Pollet, (2016) Improve Data Quality by Processing Null Values and Semantic Dependencies. Journal of Computer and Communications,04,78-85. doi: 10.4236/jcc.2016.45012